Image decoding device, image encoding device, and method thereof

ABSTRACT

The picture decoding apparatus ( 1100 ) decodes a coded stream including a moving picture stream that has pictures of different sizes with reference to decoded pictures. The picture memory ( 203 ) includes a memory area for holding the decoded pictures. The picture memory dividing method specification unit ( 210 ) divides the memory area of the picture memory ( 203 ) into picture areas of a predetermined size. The picture memory control unit ( 206 ) sequentially stores all the data of one newly decoded picture in the sequential memory areas which are formed with one or more picture areas of the picture memory ( 203 ).

TECHNICAL FIELD BACKGROUND ART

The present invention relates to a decoding method and a coding methodof a moving picture which perform an inter-picture prediction byreferring to the pictures which have been coded or decoded, inparticular, to a memory management method of the picture memory whichaccumulates decoded pictures.

DISCLOSURE OF INVENTION

Recently, with the arrival of the multimedia era in which sound, imageand other pixel values are integrated, the conventional informativemedia, such as a newspaper, a magazine, a TV, a radio and a telephone,which is a means to transmit information to people, has become a subjectfor the multimedia. In general, the multimedia refers to the expressionof not only characters, but simultaneously linked graphics, sound andespecially image. In order to make such conventional informative mediaas described above a subject for the multimedia, it is essential toexpress the information in a digital format.

However, if the information content of each informative media asdescribed above is estimated as a digital information content, thefollowing information contents will be necessary: (i) for characters,1-2 bytes per character, (ii) for sound, 64 kbits per second (telephonequality), and (iii) for moving pictures, more than 100 Mbits per second(present TV reception quality); thus it is not feasible to keep the vastamount of information of the informative media as it is in the digitalformat. For example, a TV telephone is actually used by ISDN (IntegratedServices Digital Network) which has the transmission speed of 64kbps-1.5 Mbps. However, it is impossible to transmit the image of a TVcamera as it is by ISDN.

Thus, the compressing technology of information is necessary. Forexample, in the case of a TV telephone, the moving picture compressingtechnology of H.261 and H.263 STD, internationally standardized by ITU(Telecommunication Standardization Sector of InternationalTelecommunication Union) is used. Also, according to the informationcompressing technology of MPEG-1 STD, it is possible to input imageinformation as well as sound information in a regular music CD (compactdisc).

Here, MPEG (Moving Picture Experts Group) is an international standardof the digital compressing of a moving picture signal. MPEG-1 is astandard that compresses a moving picture signal up to 1.5 Mbps, or TVsignal information to about one-hundreds. The transmission speed for aMPEG-1 STD is mainly limited to about 1.5 Mbps. MPEG-2 which isstandardized to achieve a further high image quality compresses a movingpicture signal to 2-15 Mbps.

Moreover, recently MPEG-4 which has a higher compressibility has beenstandardized by the work group (ISO/IEC JTC1/SC29/WG11) which hasdeveloped the standardization of MPEG-1 and MPEG-2. Initially, MPEG-4was not only capable of the highly efficient coding with the low bitrate, but also the powerful error tolerance technology which can lessenthe subjective image quality deterioration, even if the transmissionpath error occurs. At present the standardization activity of H.264 asthe next generation image coding formula is promoted collaboratively byISO/IEC and ITU.

Generally, a moving picture coding compresses the amount of informationby reducing the redundancy of the time direction and the spatialdirection. The inter-picture prediction coding for reducing the timeredundancy (i) refers to either of the preceding or following pictures,(ii) estimates the movement and makes a predictive picture per block,and (iii) codes the difference between the acquired predictive pictureand the present picture.

Here, a picture is a term which means a piece of image: in a progressivepicture it means a frame; and in an interlace picture it means a frameor a field. Here, an interlace picture is an image in which one frame isformed by two fields of different times. The coding and decodingprocesses of the interlace picture are capable of processing one frame(a) as a frame, (b) as two fields, and (c) as a frame structure or as afield structure per block within a frame.

Although the following picture is explained as a frame of theprogressive picture, a frame or a field of the interlace picture can beexplained in the similar manner.

FIG. 1 is a block diagram showing a structure of the conventional imagecoding apparatus 100. The image coding apparatus 100 includes: thepicture memory 101, the prediction residual coding unit 102, the codedstream generation unit 103, the prediction residual decoding unit 104,the picture memory 105, the motion vector estimation unit 106, themotion compensation coding unit 107, the motion vector memory unit 108,the picture memory control unit 109, the difference calculation unit112, the addition calculation unit 113, the switch 114 and the switch115.

The moving picture for coding is inputted into the picture memory 101per picture in the order of display. Then, the picture memory 101arranges the pictures in the order of coding. Furthermore, each pictureis divided into so called macroblocks, for example, a block ofhorizontal 16×vertical 16 pixels; and the following procedures will beperformed per block.

The input picture signal read out from the picture memory 101 isinputted into the difference calculation unit 112; and the differencepicture signal acquired by taking the difference between the inputpicture signal and the predictive picture signal outputted by the motioncompensation coding unit 107 is outputted to the prediction residualcoding unit 102. The prediction residual coding unit 102 outputs aresidual signal by performing a picture coding process such as afrequency transform or a quantization. The residual signal is inputtedinto the prediction residual decoding unit 104; and a residual decodingsignal is outputted by performing a picture decoding process such as aninverse quantization or an inverse frequency transform. The additioncalculation unit 113 generates a restructuring picture signal by addingthe residual decoding signal and the predictive picture signal; and theacquired restructuring picture signal is stored in the picture memory105.

On the other hand, the input picture signal per macroblock read out fromthe picture memory 101 is also inputted into the motion vectorestimation unit 106. Here, one piece or plural pieces of coded picturesstored in the picture memory 105 are the subject for estimation. Themotion vector which indicates the location by estimating the image areawhich is the closest to the input picture signal and the referencepicture index which indicates the selected pictures are determined. Themotion vector is estimated per block which is a unit of the furtherdivided macroblocks, and the acquired motion vector is stored in themotion vector memory unit 108. The motion compensation coding unit 107uses the motion vector and the reference picture index which areestimated by the procedures as described above. Then, the motioncompensation coding unit 107 takes out the optimum image area from thecoded pictures stored in the picture memory 105, and generates apredictive picture.

The coded stream generation unit 103 performs a variable-length codingto the coding information such as the motion vector, the referencepicture index and the residual coding signal outputted by the series ofprocedures as described above; thus the coded stream outputted by thiscoding procedures is acquired.

The procedures as described above are the operations in the case wherethe inter-picture prediction coding is performed; and these proceduresare switched with the intra-picture prediction coding by the switch 114and the switch 115. In the case of the intra picture coding, thepredictive picture is not generated by the motion compensation codingunit 107. Instead, the predictive picture of the area for coding isgenerated from the coded area within the same picture; and thedifference picture signal is generated by taking the difference. As wellas the inter-picture prediction coding, the difference picture signal isconverted to the residual coding signal in the prediction residualcoding unit 102, and variable-length coded by the coded streamgeneration unit 103. Consequently, the outputted coded stream isacquired.

FIG. 2 is a block diagram showing a structure of the conventional imagedecoding apparatus 200. The image decoding apparatus 200 includes: thecoded stream analysis unit 201, the prediction residual decoding unit202, the picture memory 203, the motion compensation decoding unit 204,the motion vector memory unit 205, the picture memory control unit 206,the addition calculation unit 211 and the switch 212.

First, the coded stream analysis unit 201 extracts each of theinformation such as the motion vector, the reference picture index andthe residual coding signal from the inputted coded stream. Each of theinformation extracted by the coded stream analysis unit 201 as describedabove is respectively outputted as following: the motion vector to themotion vector memory unit 205, the reference picture index to the motioncompensation decoding unit 204, and the residual coding signal to theprediction residual decoding unit 202.

The prediction residual decoding unit 202 performs a picture decodingoperation such as an inverse quantization and an inverse frequencytransform to the inputted residual coding signal, and outputs theresidual decoding signal. The addition calculation unit 211 adds theresidual decoding signal and the predictive picture signal outputtedfrom the motion compensation decoding unit 204. Then, the additioncalculation unit 211 generates the restructuring picture signal, andstores the acquired restructuring picture signal in the picture memory203.

The motion compensation decoding unit 204 uses the motion vectorinputted from the motion vector memory unit 205 and the referencepicture index inputted from the coded stream analysis unit 201. Then,the motion compensation decoding unit 204 takes out the optimum imagearea for the predictive picture from one or plural decoded pictureswhich are stored in the picture memory 203.

The decoded pictures generated by the series of procedures as describedabove are outputted as the picture signal for display from the picturememory 203, according to the displayed timing.

The procedures as described above are the operations performed in thecase of the inter-picture prediction decoding; and these procedures areswitched with the intra-picture prediction decoding by the switch 212.In the case of the intra-picture prediction decoding, the predictivepicture is not generated by the motion compensation decoding unit 204.Instead, (i) the predictive picture of the area for decoding isgenerated from the decoded area within the same picture, (ii) thepredictive picture and the residual decoding signal are added, and (iii)the restructuring picture signal is generated. The acquiredrestructuring picture signal is stored in the picture memory 203, andoutputted as the picture signal for display, according to the displayedtiming.

(a) and (b) in FIG. 3 are the diagrams for explaining the referencerelations of the pictures in the inputted image. (a) in FIG. 3 shows thepictures arranged in the order of display; and (b) in FIG. 3 shows thepictures arranged in the order of coding. The number added to “P” ofeach picture shows the number of the display order for the picture. Suchnumber as described above is given to each picture so that all thenumbers are in the rising order when each picture is arranged in theorder of display as in (a) in FIG. 3. Also, the arrows in FIG. 3 showthe reference relations among the pictures in the inter-pictureprediction coding or the inter-picture prediction decoding; and thepicture of the root of the arrows can arbitrarily refer to one or two ofthe pictures of the tip of the arrows.

For example, P4 performs coding or decoding referring to P0, P1, P2 andP6. The plural pictures used for the reference need to be coded ordecoded before the picture P4, and be stored as the reference picturesin the picture memory. As in (b) in FIG. 3 where the pictures arearranged in the order of coding, it is noticeable that all the picturesreferred by P4 are located in front of P4.

In the procedures of coding and decoding, the picture P4 selects one ortwo pieces which are the optimum among the plural pictures used for thereference per block, and uses the selected pictures as the referencepictures for the inter-picture prediction coding and the inter-pictureprediction decoding which will be explained later.

Each picture stored in the picture memory is attached with theinformation that indicates whether or not it can be used for reference.The picture that is not used for reference should be immediately deletedfrom the picture memory; however, even if it is known that the pictureis not used for reference, it needs to wait for the other pictures to bedecoded until its turn to be displayed comes. For example, P3 of (b) inFIG. 3 is the second to be decoded; however, it is the fourth to bedisplayed in (a) in FIG. 3. Thus, P3 has to be stored in the picturememory until P1 and P2 are decoded. The picture such as P3 is called the“display-waiting picture” or the “non-reference picture”. The picturethat can be used for reference is called the “reference picture”.

Generally, the picture signals restructured in (a) the additioncalculation unit 113 of the image coding apparatus 100, and (b) theaddition calculation unit 211 of the image decoding apparatus 200 arerespectively accumulated in the picture memory 105 and the picturememory 203 under the control of the picture memory control unit 109 andthe picture memory control unit 206 as described below. FIG. 4 is a flowchart showing the hold controlling operation of the reconstructedpicture signal into the picture memory 203 in the image decodingapparatus 200. When the current picture is decoded (S401) and the newlyreconstructed picture signal is generated, the picture memory controlunit 206 judges whether or not the picture memory 203 has enough freearea to hold the generated reconstructed picture signal (S402). If thepicture memory 203 does not have enough free area to hold the generatedreconstructed picture signal, the picture memory control unit 206deletes the oldest picture in the order of display, already stored inthe picture memory 203 (S403). In this manner the picture memory controlunit 206 sequentially deletes the already stored pictures from theoldest one in the order of display by repeating the step S402 and thestep S403 until the picture memory 203 has enough free area to holdnewly reconstructed picture signal. Thus, when the picture memory 203has enough free area to hold the newly reconstructed picture signal, itstores the newly reconstructed picture signal in the free area (S404).Here, the free area means the area that enables the corresponding areawithin the memory to be reused by other data. Also, deleting means theprocedure to enable the use of the memory area for accumulating otherdata; and it can be performed in the same manner both in the case ofphysically erasing the data and allowing only to overwrite withoutphysically erasing the data. (refer to “Joint Final Committee Draft(JFCD) of Joint Video Specification (ITU-T Rec. H.264|ISO/IEC 14496-10AVC)” 7.4.3.3 P.65, E.2.1 P.252)

FIG. 5A, FIG. 5B, FIG. 6A, FIG. 6B, FIG. 7A and FIG. 7B are the diagramsshowing the examples of the use of the memory area in the case where thepicture memory is managed according to the procedures as shown in FIG.4.

FIG. 5A is a diagram showing the holding example of a picture to bedecoded in the case where the picture size is changed from small tolarge and the picture memory 203 has enough free area to hold the largepicture. The head part of FIG. 5A shows the situation where the picturememory 203 has enough free area (the oblique lined part) to accumulatethe decoded picture C. This example shows the situation where P2, P6, P4and P5 of the picture stream of FIG. 3 are accumulated in the picturememory, and P7 (=the picture C) is decoded. In such case as describedabove, as shown in the bottom part of FIG. 5A, it is possible toaccumulate the decoded picture C as it is in the head of the free areaof the picture memory 203. In other words, the pictures that are alreadyaccumulated in the picture memory are not deleted.

FIG. 5B shows a diagram of a holding example of a picture to be decodedin the case where the picture size is changed from small to large, andthe picture memory 203 does not have enough free area to hold the largepicture. In FIG. 5B the pictures with the numbers circled with O are thepictures that should be deleted from the picture memory 203 in order tonewly hold the picture C. The head part of FIG. 5B shows the situationwhere the picture memory 203 does not have enough free area toaccumulate the decoded picture C. This example shows the situation whereP3, P1, P2, P6, P4 and P5 of the picture stream of FIG. 3 areaccumulated in the picture memory, and P7 is decoded at present (thatis, C=P7). In such case as described above, it is necessary to retain anarea to accumulate the newly decoded picture C by deleting the picturesaccumulated in the picture memory 203 before the picture size ischanged. In this example, it is noticeable that the picture C, after thechange of the picture size, is larger than that of prior-change of thepicture size, and the two pictures that are already stored need to bedeleted in order to newly hold the picture C. Next, the picture to bedeleted is sequentially selected from the oldest one of the displayorder information that each picture has. In this example, the displayorder information of P1 is the oldest in the picture memory 203, and itis noticeable that the display order information becomes newer in theorder of P1, P2, P3 . . . . The picture memory control unit 206 judgesone or plural pictures to be deleted according to such procedures asdescribed above. Then, the picture memory control unit 206 outputs thejudging result to the picture memory 203, and deletes the picture. Inthe example as shown in the head part of FIG. 5B, P1 and P2 are deletedby such procedures as described above; and the decoded picture C isaccumulated in the head part of the newly generated free area as shownin FIG. 5B.

FIG. 6A shows a diagram of a holding example of a picture to be decodedin the case where (i) the picture size is not changed, (ii) the picturethat has a different picture size from the picture to be decoded doesnot exist in the picture memory, and (iii) the picture memory 203 hasenough free area to hold the newly decoded picture C. In other words, itis an example that probably happens in the case where the picture sizeis not changed in the temporal proximity of the picture C, or the streamwhere the change of the picture size does not happen is decoded. In suchcase as described above, the pictures that are already stored in thepicture memory 203 and the newly decoded picture C all have the samepicture size. Thus, in order to hold the newly decoded picture C, onlyone of the pictures is deleted from the picture memory 203. Therefore,the picture memory control unit 206 checks the free area of the picturememory 203, and makes the following judgments: (a) to write the pictureC in if there is enough free area to hold the picture C, and (b) todelete one picture if there is not enough free area to hold the pictureC. The pictures accumulated in the picture memory 203 have theinformation indicating the display order, in addition to the picturedata. The picture memory control unit 206 checks the display orderinformation. Then, the picture memory control unit 206 judges thepicture to be deleted, and transmits it as the judging result to thepicture memory 203 that only one picture whose display order informationis the oldest should be deleted. As a result, the decoded picture C isaccumulated in the generated free area.

The head part of FIG. 6A shows the situation where the picture memory203 has enough free area to accumulate the newly decoded picture C. Thisexample shows the situation where P2, P6, P4 and P5 of the picturestream of FIG. 3 are accumulated in the picture memory and P7 is decodedat present (that is, C=P7). In such case as described above, as shown inthe bottom part of FIG. 6A, it is possible to accumulate the picture Cas it is in the free area. In other words, the pictures that are alreadyaccumulated in the picture memory are not deleted.

FIG. 6B shows a holding example of a picture to be decoded in the casewhere (i) the picture size is not changed, (ii) the picture that has adifferent picture size from the picture to be decoded does not exist inthe picture memory, and (iii) the picture memory 203 does not haveenough free area to hold the newly decoded picture C. The head part ofFIG. 6B shows the situation where the picture memory does not haveenough free area to accumulate the decoded picture C. This example showsthe situation where P3, P1, P2, P6, P4 and P5 of the picture stream inFIG. 3 are accumulated in the picture memory, and P7 is decoded atpresent (that is, C=P7). In such case as described above, it isnecessary to retain an area to accumulate the newly decoded picture C bydeleting only one of the pictures accumulated in the picture memory 203before the picture size is changed. The picture to be deleted issequentially selected from the oldest one of the display orderinformation that each picture has. In this example, it is noticeablethat the display order information of P1 is the oldest in the picturememory 203. The picture memory control unit 206 judges the picture to bedeleted according to such procedures as described above. Then, thepicture memory control unit 206 outputs the judging result to thepicture memory 203, and deletes the picture. In the example as shown inthe head part of FIG. 6B, P1 is deleted by such procedures as describedabove; and the decoded picture C is accumulated in the newly generatedfree area as shown in the bottom part of FIG. 6B.

FIG. 7A shows a diagram of a holding example of a picture to be decodedin the case where (i) the picture size is not changed, (ii) the picturethat has a different picture size from the picture to be decoded existsin the picture memory, and (iii) the picture memory 203 has enough freearea to hold the newly decoded picture C. In other words, it is anexample that probably happens in the case where the picture to bedecoded is decoded before a long time passes after the picture size ischanged. The picture memory control unit 206 checks the free area of thepicture memory 203. Then, the picture memory control unit 206 does notchange anything if there is enough free area to hold the newly decodedpicture C. And, the picture memory control unit 206 judges the number ofthe pictures to be deleted and the pictures to be deleted, by comparing(a) the size of the picture C that is newly decoded after the change ofthe picture size and (b) the size of the picture C before the change ofthe picture size, if there is not enough free area to hold the pictureC. In the case where the picture memory control unit 206 (i)sequentially deletes only the necessary pieces of pictures from the onewhose display order information is the oldest, and (ii) stores the newlydecoded picture C in the newly generated free area, the picture memorycontrol unit 206 specifies the picture to be deleted by checking thedisplay order information as described above. The picture to be deletedis transmitted from the picture memory control unit 206 to the picturememory 203. As a result, the newly decoded picture C is accumulated inthe generated free area.

The head part of FIG. 7A shows the situation where the picture memory203 has enough free area to accumulate the newly decoded picture C. Thisexample shows the situation where P4, P5, and P7 of the picture streamof FIG. 3 are accumulated in the picture memory and P8 is decoded atpresent (that is, C=P8). In such case as described above, as shown inthe bottom part of FIG. 7A, it is possible to accumulate the picture Cas it is in the free area. In other words, the pictures that are alreadyaccumulated in the picture memory are not deleted.

FIG. 7B shows a holding example of a picture to be decoded in the casewhere (i) the picture size is not changed, (ii) the pictures that have adifferent picture size from the picture to be decoded exists in thepicture memory, and (iii) the picture memory 203 does not have enoughfree area to hold the newly decoded picture C. The head part of FIG. 7Bshows the situation where the picture memory does not have enough freearea to accumulate the newly decoded picture C. This example shows thesituation where P2, P6, P4, P5, and P7 of the picture stream in FIG. 3are accumulated in the picture memory, and P8 is decoded at present(that is, C=P8). In such case as described above, it is necessary toretain an area to accumulate the newly decoded picture C by deleting thepictures accumulated in the picture memory 203 before the picture sizeis changed. In this example, it is noticeable that the picture C, afterthe change of the picture size, is larger than the picture C, before thechange of the picture size; and the two pieces of pictures need to befurther deleted, even if the already free area for holding the picture Cis used. Next, the picture to be deleted is sequentially selected fromthe oldest one of the display order information that each picture has.In this example, it is noticeable that the display order information ofP2 is the oldest; and the display order information becomes newer in theorder of P2, P4, P5 . . . . Thus, the picture memory control unit 206judges one or plural pictures to be deleted according to such proceduresas described above. Then, the picture memory control unit 206 outputsthe judging result to the picture memory 203, and deletes the picture.In the example as shown in the head part of FIG. 7B, P2 and P4 aredeleted by such procedures as described above; and the newly decodedpicture C is accumulated in the newly generated free area as shown inthe bottom part of FIG. 7B.

However, according to the decoding method explained with theconventional technology, because the pictures to be deleted, P2 and P4are stored in the physically separate areas within the picture memory203, as happened in the example of the bottom part of FIG. 7B, thesituation where the newly decoded picture is divided and accumulated inthe physically separate areas occurs. In particular, depending on thecombination of the picture sizes of the pictures which are included inthe stream for coding, there is a probability that one picture isdivided into different areas in a very complicated way. In such casewhere the pictures that are accumulated in the physically separate areasare used as reference pictures as described above, the access to thepicture memory becomes complicated, and there is a probability that thedecrease of the access speed and the decoding speed is caused.

An object of the present invention, in view of the above problem, is toprovide the management method of the picture memory that prevents theefficiency of the access to the picture memory from decreasing, in thecase where the picture size may change at the beginning of or in themiddle of the moving picture stream.

DISCLOSURE OF INVENTION

The picture decoding apparatus according to the present invention fordecoding a coded stream including a moving picture stream that haspictures of different sizes, with reference to a decoded picture,comprises: a picture memory including a memory area for holding decodedpictures; a memory division unit operable to divide the memory area ofthe picture memory into division areas of a predetermined size; and astorage unit operable to sequentially store all data of one newlydecoded picture in sequential memory areas which are formed with one ormore of the division areas of said picture memory.

Thus, it is possible to provide the picture decoding apparatus thatprevents the efficiency of the access to the picture memory fromdecreasing, in the case where the picture size may change at thebeginning of or in the middle of the moving picture stream.

Also, such picture decoding apparatus as described above furthercomprises a size change judgment unit operable to judge whether or not apicture size has been changed by comparing a picture size of a newlydecoded picture and a picture size of a picture just decoded before,wherein in the case where it is judged that the picture size of thenewly decoded picture has been changed, said memory division unitdivides the memory area so that the size of the division area becomesthe picture size of the newly decoded picture.

Thus, it is possible to provide the management method of the picturememory that prevents the efficiency of the access to the picture memoryfrom decreasing, in the case where the picture size may change at thebeginning of or in the middle of the moving picture stream.

Moreover, such picture decoding apparatus as described above furthercomprises a reading unit operable to read division informationindicating a division method of the memory area from the coded stream,wherein said memory division unit divides the memory area according tothe read division information.

Thus, it is possible to realize the management method of the picturememory that prevents the efficiency of the access to the picture memoryfrom decreasing without significantly increasing the throughput of thepicture decoding apparatus.

Furthermore, the present invention can be realized as a moving picturecontent delivery system which distributes coded moving picture contentsfrom a distribution server including a coding apparatus for movingpictures and reproduces the distributed moving picture contents in auser's side terminal apparatus including a decoding apparatus for movingpictures. In addition, the present invention is capable of realizing thefollowing: (a) individual units such as a distribution server and auser's side terminal apparatus which form a moving picture contentdelivery system, (b) a coding apparatus and a decoding apparatusincluded in the distribution server and the user's side terminal, (c) acoding method and a decoding method which have characteristic units, assteps, with which the moving picture content delivery system isequipped, and (d) a program which has a computer perform such steps asdescribed above. Needless to say, such program as described above can beprovided via a record medium such as a CD-ROM or a transmission mediumsuch as the Internet.

Further Information about Technical Background to this Application

The disclosure of Japanese Patent Application No. 2003-040804 filed onFeb. 19, 2003 including specification, drawings and claims isincorporated herein by reference in its entirety.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing a structure of the conventionalpicture coding apparatus;

FIG. 2 is a block diagram showing a structure of the conventionalpicture decoding apparatus;

-   -   (a) and (b) in FIG. 3 are diagrams explaining reference        relations of the pictures among the inputted pictures. (a) in        FIG. 3 shows a situation where the pictures are arranged in the        display order. (b) in FIG. 3 shows a situation where the        pictures are arranged in the order of coding;

FIG. 4 is a flow chart showing a holding control operation of therestructuring picture signal into the picture memory in the picturedecoding apparatus as shown in FIG. 2;

FIG. 5A is a diagram showing a holding example of a picture for decodingin the case where the picture size is changed from small to large, andthe picture memory has enough free area to hold the enlarged picture.FIG. 5B is a diagram showing a holding example of a picture for decodingin the case where the picture size is changed from small to large, andthe picture memory does not have enough free area to hold the enlargedpicture;

FIG. 6A is a diagram showing a holding example of a picture for decodingin the case where (i) the picture size is not changed, (ii) the picturememory does not have any pictures of different size from the picture tobe decoded, and (iii) the picture memory has enough free area to holdthe newly decoded picture C. FIG. 6B is a diagram showing a holdingexample of the picture to be decoded in the case where (i) the picturesize is not changed, (ii) the picture memory does not have any picturesof different size from the picture for decoding, and (iii) the picturememory does not have enough free area to hold the newly decoded pictureC;

FIG. 7A is a diagram showing a holding example of the picture fordecoding in the case where (i) the picture size is not changed, (ii) thepicture memory has the pictures of different size from the picture C,and (iii) the picture memory has enough free area to hold the newlydecoded picture C. FIG. 7B is a diagram showing a holding example of thepicture to be decoded in the case where (i) the picture size is notchanged, (ii) the picture memory has the pictures of different size fromthe newly decoded picture C, and (iii) the picture memory does not haveenough free area to hold the newly decoded picture C;

FIG. 8 is a block diagram showing a structure of the picture codingapparatus according to the first embodiment;

FIG. 9A and FIG. 9B are diagrams showing a data structure of a codedstream generated by the picture coding apparatus 800 according to thepresent embodiment. FIG. 9A is a diagram showing an example of the codedstream in the case where the information of the biggest picture sizewhich may be included in the stream is coded. FIG. 9B is a diagramshowing an example of the coded stream in the case where the picturememory is divided based on the biggest picture size which may beincluded in the stream, and the number of areas which can be included inthe picture memory is coded;

FIG. 10A and FIG. 10B are diagrams showing an example of the codedstream in the case where additional information is written in the areaof the picture common information of the coded stream shown in FIG. 9Aand FIG. 9B;

FIG. 11 is a block diagram showing a structure of the image decodingapparatus of the present embodiment;

FIG. 12 is a flow chart showing the order of decoding procedures of theimage decoding apparatus as shown in FIG. 11;

FIG. 13A and FIG. 13B are diagrams showing a situation where the newlydecoded picture is accumulated according to the order of the stepsS1204-S1210 of the flow chart as shown in FIG. 12. FIG. 13A is anexample of the case where the picture memory has enough free area toaccumulate the decoded picture C. FIG. 13B is an example of the casewhere the picture memory does not have enough free area to accumulatethe decoded picture C;

FIG. 14A and FIG. 14B are diagrams showing a situation where the newlydecoded picture is accumulated in the picture memory according to theorder of the steps S1204-S1213 of the flow chart as shown in FIG. 12.FIG. 14A is an example of the case where the picture memory has enoughfree area to accumulate the decoded picture C. FIG. 14B is an example ofthe case where the picture memory does not have enough free area toaccumulate the decoded picture C;

FIG. 15A, FIG. 15B and FIG. 15C are diagrams showing a data structure ofa coded stream generated by the picture coding apparatus of the secondembodiment. FIG. 15A is a diagram showing an example of the coded streamin the case where the information of the smallest picture size which maybe included in the stream is coded. FIG. 15B is a diagram showing anexample of the coded stream in the case where the size information ofthe picture area which is acquired by dividing the picture memory iscoded. FIG. 15C is a diagram showing an example of the coded stream inthe case where the number of areas in dividing the area is coded;

FIG. 16A is a diagram showing an example of three kinds of picture sizeswhich are coded within one stream. FIG. 16B is a diagram showing anexample of dividing method of the picture area for efficiently using thepicture memory in the case where the pictures of the three kinds ofpicture sizes as shown in FIG. 16A are included within one stream;

FIG. 17A, FIG. 17B and FIG. 17C are diagrams showing an example of thecoded stream in the case where additional information is written in thearea of the picture common information of each coded stream as shown inFIG. 15A, FIG. 15B and FIG. 15C;

FIG. 18A and FIG. 18B are diagrams showing a situation where the newlydecoded pictures are accumulated in the picture memory according to theorder of the steps S1204-S1210 of the flow chart of FIG. 12;

FIG. 19A and FIG. 19B are diagrams showing a situation where the newlydecoded pictures are accumulated in the picture memory according to theorder of the steps S1204-S1213 of the flow chart of FIG. 12;

FIG. 20 is a block diagram showing a structure of the picture decodingapparatus according to the third embodiment of the present invention;

FIG. 21 is a flow chart showing the order of the decoding procedures ofthe picture decoding apparatus as shown in FIG. 20;

FIG. 22A and FIG. 22B are diagrams showing a situation where the newlydecoded picture is accumulated in the picture memory according to theorder of the flow chart as shown in FIG. 21;

FIG. 23A and FIG. 23B are diagrams showing the situation where the newlydecoded picture is accumulated in the picture memory according to theorder of the flow chart as shown in FIG. 21;

FIG. 24 is a flow chart showing the order of the other decodingprocedures of the picture decoding apparatus as shown in FIG. 20;

FIG. 25 is a flow chart showing the more specific order of theprocedures of the step S2605 as shown in FIG. 24;

FIG. 26A and FIG. 26B are diagrams showing a situation where the newlydecoded picture is accumulated in the picture memory according to theorder of the flow chart of FIG. 24 and FIG. 25;

FIG. 27A and FIG. 27B are diagrams showing a situation where the newlydecoded picture is accumulated in the picture memory according to theorder of the flow chart of FIG. 24 and FIG. 25;

FIG. 28 is a block diagram showing a structure of the picture codingapparatus according to the fourth embodiment;

FIG. 29 is a diagram showing a data structure of the coded streamgenerated by the picture coding apparatus according to the fourthembodiment;

FIG. 30A shows an example of a physical format of a flexible disc whichis a record medium itself. FIG. 30B shows the appearance of the flexibledisc seen from the front, a section structure, and the flexible disc.FIG. 30C shows a structure for playing the record of such program asdescribed above with the flexible disc FD;

FIG. 31 is a block diagram showing a whole structure of a content supplysystem ex100 which realizes a content delivery service;

FIG. 32 is a diagram showing a cellular phone ex115 which uses suchmoving picture coding method and the moving picture decoding method asexplained in the embodiments;

FIG. 33 is a block diagram showing a structure of a cellular phone; and

FIG. 34 is a diagram showing an example of a digital broadcastingsystem.

BEST MODE FOR CARRYING OUT THE INVENTION

The embodiments of the present invention will be explained using FIG.8-FIG. 34 as following.

First Embodiment

Conventionally the picture memory control (holding and deleting apicture in the memory area) when decoding was performed per picture. Onthe other hand, according to the present embodiment, a memory area of apicture memory when decoding is divided into one or plural areas; and adivided area is a unit for controlling the picture memory. According tothe present embodiment, information for specifying a dividing method ofthe picture memory is generated in the picture coding apparatus, andwritten in a coded stream. Here, the divided area is called a picturearea.

A coding method of a moving picture according to the first embodimentwill be explained using a block diagram as shown in FIG. 8. FIG. 8 is ablock diagram showing a structure of a picture coding apparatus 800 ofthe first embodiment. The structure of the picture coding apparatus 800is similar to that of the conventional picture coding apparatus 100 asexplained in FIG. 1. The difference from the conventional picture codingapparatus 100 is that a picture memory dividing information coding unit801 is newly added. The same procedures as the conventional technologywill be omitted here.

Preceding the coding of a stream for coding, the picture memory dividinginformation coding unit 801 codes the information for specifying themethod to divide the picture memory of the decoding side. FIG. 9A andFIG. 9B are diagrams showing a data structure of a coded streamgenerated by the picture coding apparatus 800. FIG. 9A is a diagramshowing an example of a coded stream in the case where the informationof the biggest picture size which may be included in the stream iscoded. FIG. 9B is a diagram showing an example of a coded stream in thecase where the picture memory is divided based on the biggest picturesize which may be included in the stream, and the number of the areasthat can be stored in the picture memory is coded. According to thefirst embodiment, as the information for specifying the method to dividethe picture memory of the decoding side, the information of the biggestpicture size which may be included in the stream is coded; and the codedinformation is written as max_size of the picture common informationarea of the coded stream as shown in FIG. 9A. Here, for example, thebiggest picture size that the picture coding apparatus 800 can use forcoding is used. Otherwise, when a user performs coding, he can selectand determine the desirable biggest picture size among the suggestedplural picture sizes.

In the case where the picture for coding is capable of changing thepicture size of an IDR picture or an equivalent picture thereto, thepicture memory control unit 109 inputs a signal showing a reset methodof the picture memory into the coded stream generation unit 103. An IDRpicture means a picture that has special characteristics as following:(i) a picture which is capable of only intra-picture predictiondecoding; (ii) a picture which resets all the pictures in the picturememory after decoding is finished; and (iii) a picture which is capableof changing the picture size. Thus, a picture which is decoded after anIDR picture cannot refer to a picture which is decoded before an IDRpicture. For example, P8 as shown in (a) in FIG. 3 can only refer to P7.An IDR picture has reset_flag in the header area of the slice data asshown in FIG. 9A, and the reset_flag specifies the reset method of thepicture memory which is one of the characteristics of an IDR picture.Here, a slice means each of the divided areas when a picture is dividedinto one or plural areas. When reset_flag is 0, all the “referencepictures” in the picture memory are changed to “non-reference pictures”(“display-waiting pictures”). On the other hand, when reset flag is 1,regardless of “reference pictures” or “non-reference pictures”, the dataof all the pictures is deleted from the picture memory. In other words,when reset_flag is 1, even if there are pictures that are decoded beforean IDR picture and not displayed yet, the pictures after an IDR pictureare displayed without displaying such pictures as described above.According to the first embodiment, the reset method of the picturememory is arbitrarily specified. For example, in the operation of thepicture memory of the decoding side, depending on whether or not thereare pictures that are not displayed in the picture memory, “reset_flagis set 0”, that is, all the pictures are made “non-reference pictures”;and the picture data itself is not deleted. Otherwise, it is possible toarbitrarily select either of the two: (a) “reset_flag is set 0” or (b)“reset_flag is set 1”, that is, all the picture data itself is deleted;and all the area of the picture memory is made empty. The coded streamgeneration unit 103 codes the inputted signal, and generates a codedstream which has reset_flag in the slice header as shown in FIG. 9A.

Also, instead of writing the information of the biggest picture sizewhich may be included in the stream, it is possible to write the numberof the areas that can be retained in the picture memory when decoding asmin_area_num in the picture common information area of the coded stream,as shown in FIG. 9B. min_area_num indicates the number of the pictureareas that are made when dividing the picture memory on the decodingside so that the size of one picture area becomes max_size that is thebiggest picture size that may be included in the stream. In such case asdescribed above, the value of min_area_num is determined by theequations as shown below. Here, the calculation result of the divisionin the equation is calculated as an integer number by round down method.(min_area_num)=(the picture memory size on the decodingside)/(max_size)  (Equation 1)

Also, it is possible to treat the picture size and picture memory sizein the same manner, no matter which of the followings they may be: thewhole number of pixels, the whole number of macroblocks and the wholenumber of bits.

Also, in inter-picture prediction coding, by performing the coding withsuch number of the “reference pictures” as is always lower thanmin_area_num which is calculated by the equation 1 as shown above,regardless of the picture size of the pictures for coding, even if thedivided picture memory is used by the dividing method as described aboveon the decoding side, it is possible to prevent the situation where thereference pictures do not exist in the picture memory from happening.

FIG. 10A and FIG. 10B are diagrams showing an example of the codedstream in the case where additional information is written in the areaof the picture common information within the coded stream as shown inFIG. 9A and FIG. 9B. Also, it is possible to write cnf_flag, in additionto max_size and min_area_num, in the picture common information area ofthe coded stream as shown in FIG. 10A and FIG. 10B. Even in the casewhere the picture memory on the decoding side is divided and controlled,according to max_size and min_area_num, cnf_flag indicates whether ornot all the pictures included in the stream can be displayed without anyproblems. For example, in the case where the value of cnf_flag is 1, itindicates that all the pictures in the stream can be displayed withoutany problems. In the case where the value of cnf_flag is 0, it indicatesthe possibility that the pictures that are not displayed yet may not bedisplayed because the data is already lost, due to the procedures of thefirst embodiment. Thus, by writing cnf_flag in the coded stream andchecking the value of this signal, it is possible to judge whether ornot the procedures of the first embodiment should be applied. Forexample, if cnf_flag is 0, it should be noted that there is apossibility that some pictures may not be displayed if the procedures ofthe first embodiment are applied. Therefore, by switching to theconventional method of controlling the picture memory per picture, withthe possibility that the memory area may be fragmented, all the picturescan be displayed without any problems.

Although in FIG. 9A, FIG. 9B, FIG. 10A and FIG. 10B, max_size,min_area_num and cnf_flag are written in the picture common informationarea, it is also possible to write max_size, min_area_num and cnf_flagin the datan area where the data that is not essential, but supports thedecoding procedures is collected. Also, it is possible to writemax_size, min_area_num and cnf_flag in the sequence common informationarea that can be referred to by all the sequences. Moreover, it ispossible to write max_size, min_area_num and cnf_flag in the sliceheader.

Next, the decoding method of the coded stream which is coded by suchcoding method as described above will be explained using FIG. 11-FIG.14. FIG. 11 is a block diagram showing a structure of the picturedecoding apparatus 1100 of the first embodiment. FIG. 12 is a flow chartshowing the order of the decoding procedures in the picture decodingapparatus 1100 as shown in FIG. 11. FIG. 13A, FIG. 13B, FIG. 14A andFIG. 14B are schematic diagrams showing the management method of thepicture memory 203 as shown in FIG. 11.

The structure of the picture decoding apparatus 1100 as shown in FIG. 11is similar to that of the conventional picture decoding apparatus 200 asshown in FIG. 2. The difference from the conventional method is that thepicture size change judgment unit 207, the deletable picture areajudgment unit 209 and the picture memory dividing method specificationunit 210 are added. The same procedures as the conventional technologywill be omitted here.

Unlike the conventional technology, in the first embodiment, the picturememory 203 is not controlled per picture; however, the picture area thatis made by dividing the picture memory 203 into one or plural areas isused as a unit for holding and deleting a picture. Preceding thedecoding of the coded stream, the picture memory dividing methodspecification unit 210 analyzes the dividing method of the picturememory from a signal of the coded stream, and directs the picture memorycontrol unit 206. According to the first embodiment, as the informationfor specifying the dividing method of the picture memory, theinformation of the biggest picture size that may be included in thestream is written in the picture common information area as max_sizethat is included in the stream of FIG. 9A. The picture memory dividingmethod specification unit 210 analyzes the signal, and determines thedividing method of the picture memory 203. The number of the areasdivided is calculated by the equation 2 as shown below. Also, thecalculation result of the division in the equation is calculated as aninteger number by the round down procedure.(The number of the picture areas)=(The size of the picture memory 203 ofthe picture decoding apparatus 1100)/(max_size)  (Equation 2)

Also, in the case where the coded stream of FIG. 9B is used in stead ofthe coded stream of FIG. 9A, the number of the picture areas iscalculated by the equation 3 as shown below.(The number of the picture areas)=(min_area_num)  (Equation 3)

Here, the picture size and the picture memory size as described abovecan be treated in the same manner, no matter which of the three they maybe: the whole number of pixels, the whole number of macroblocks and thewhole number of bits.

The picture size change judgment unit 207 compares the picture size ofthe picture to be decoded and the picture size of the previously (orjust) coded picture, and judges whether or not the picture size has beenchanged.

reset_flag is coded in the slice header of the coded stream, as shown inFIG. 9A and FIG. 9B. The coded stream analysis unit 201 analyzes thevalue of the reset_flag, and outputs the information of the analysisresult to the picture memory control unit 206. According to the firstembodiment, even if the picture size was changed, there is a possibilitythat the value of reset_flag is selected as either 0 or 1. Thus,according to the value of reset_flag, the management method of thepicture memory is divided as shown in the flow chart of FIG. 12. Inparticular, in the case where reset_flag is 0, there is a possibilitythat the picture before the change of the picture size remains in thepicture memory; therefore, it is necessary to judge which picture areashould be deleted in order to accumulate the picture after the change ofthe picture size. The deletable picture area judgment unit 209 makessuch judgment as described above; and the judging result is outputted tothe picture memory control unit 206. Consequently, the picture memory ismanaged.

Next, the specific operations of the picture memory control unit 206,the picture size change judgment unit 207 and the deletable picture areajudgment unit 209 in the decoding procedure will be explained using theflow chart of FIG. 12. Although in the flow chart in FIG. 12, the changeof the picture size and the value of reset_flag are judged after thecurrent picture is decoded, such order is arbitrary and not limited tothe order as shown in FIG. 12.

First, the flow of the decoding procedure will be explained in the casewhere the picture size of the picture to be decoded is changed (S1204)and reset_flag is 1 (S1205). The value of reset_flag as 1 indicates thatall the picture data in the picture memory is deleted and all the areais emptied. In other words, if such signal as described above isinputted into the picture memory control unit 206, it is possible thatthe picture memory control unit 206 deletes all the areas of the picturememory (S1206), the decoded picture after the change of the picture sizeis saved in the arbitrary area in the picture memory (S1207).

Next, the flow of the decoding procedure will be explained in the casewhere the picture size of the picture to be decoded is changed (S1204),and reset_flag is 0 (S1205). The value of reset_flag as 0 indicates thatall the picture data in the picture memory is made “non-reference”pictures, but the picture data itself is not deleted and leftaccumulated. The deletable picture area judgment unit 209 checks thefree area of the picture memory, and judges the picture area(s) to bedeleted in the case where there is not enough free area (S1208). Thepictures that are accumulated in the picture memory 203 have the displayorder information, in addition to the picture data. The deletablepicture area judgment unit 209 checks the display order information.Then, the deletable picture area judgment unit 209 specifies the picturearea where the picture that has the oldest display order information isaccumulated, and transmits the direction to delete the specified picturearea to the picture memory control unit 206. Consequently, the specifiedpicture area is deleted, and the decoded picture is accumulated in thegenerated free area (S1210).

FIG. 13A and FIG. 13B are diagrams showing a situation where the newlydecoded picture is accumulated in the picture memory 203, following theorder of the steps S1204-S1210 of the flow chart as shown in FIG. 12.

FIG. 13A is an example of the case where there is enough free area toaccumulate the decoded picture C in the picture memory 203. In suchexample as described above, P4 and P5 of the picture stream in (a) and(b) in FIG. 3 are accumulated in the picture memory, and P7 is decodedat present (that is, C=P7). P4 and P5 are the pictures that have thesame picture size as the divided picture area; thus, it should be notedthat P4 and P5 have the biggest picture size among the pictures includedin the sequence. On the other hand, the decoded picture C always has thepicture size that is the same as or smaller than the picture area. Theexample of FIG. 13A indicates the case where the picture size of thepicture C for decoding is smaller than P4 and P5. In such case asdescribed above, it is possible to accumulate the decoded picture C inthe free area as it is as shown in the bottom part of FIG. 13A. In otherwords, the picture area where the picture data is already accumulated isnot deleted.

The head part of FIG. 13B is an example of the case where there is notenough free area to accumulate the decoded picture C in the picturememory. In such example as described above, P6, P4 and P5 of the picturestream as shown in FIG. 3 are accumulated in the picture memory, and P7is decoded at present (that is, C=P7). In such case as described above,it is necessary to delete the picture area that has the picture databefore the change of the picture size and is accumulated in the picturememory 203, and to retain an area for accumulating the decoded pictureC. According to the first embodiment, the picture to be decoded alwayshas the picture size that is the same as or smaller than the picturearea. Thus, in the case where the picture memory 203 does not have afree area, the area for accumulating the decoded picture can be retainedby deleting one of the picture areas. The picture area where theaccumulated picture has the oldest display order information is selectedas the picture area to be deleted. In such example as described above,it should be noted that the display order information of P4 is theoldest. The deletable picture area judgment unit 209 judges the picturearea to be deleted by such procedures as described above. Then, thedeletable picture area judgment unit 209 outputs the judging result tothe picture memory control unit 206, and deletes the picture area. Inthe example of the head part of FIG. 13B, P4 is deleted by suchprocedures as described above; and the decoded picture C as shown in thebottom part of FIG. 13B is accumulated in the free area which isgenerated by deleting P4.

Finally, the flow of the decoding procedure will be explained in thecase where the picture size of the picture to be decoded is not changedwhen decoding, but is already changed, or in the case where the picturesize of the picture is not changed at all (S1204). Unlike theconventional technology, according to the first embodiment, regardlessof the existence of the picture whose size is different from the pictureC for decoding in the picture memory 203, it is possible to manage thepicture memory 203 by the same procedure as described above. Thedeletable picture area judgment unit 209 checks the free area of thepicture memory 203, and judges the picture area to be deleted in thecase where there is not enough free area. The pictures that areaccumulated in the picture memory 203 have the display orderinformation, in addition to the picture data. The deletable picture areajudgment unit 209 checks the display order information. Then, thedeletable picture area judgment unit 209 specifies the picture areawhere the picture whose display order information is the oldest isaccumulated, and transmits the direction to delete the specified picturearea to the picture memory control unit 206. As a result, the deletablepicture area judgment unit 209 accumulates the decoded picture in thegenerated free area.

FIG. 14A and FIG. 14B are diagrams showing a situation where the newlydecoded picture is accumulated in the picture memory 203 following thesteps S1204-S1213 of the flow chart as shown in FIG. 12.

FIG. 14A is an example of the case where the picture memory has enoughfree area to accumulate the decoded picture C. Such example as describedabove shows the situation where P5 and P7 of the picture stream areaccumulated in the picture memory and P8 is decoded (that is, C=P8) atpresent. P5 has the same picture size as the picture area. Thus, itshould be noted that P5 has the biggest picture size among the picturesincluded in the sequence. On the other hand, the decoded picture Calways has the same picture size as or smaller picture size than thepicture area. The example of FIG. 14A shows the case where the picture Chas a smaller picture size than P5. In such case as described above, asshown in the bottom part of FIG. 14A, it is possible to accumulate thedecoded picture C as it is in the free area. However, in such case asdescribed above, the picture C for decoding is not accumulated from thehead of the free area; but, the picture C is accumulated from the headof the picture area that is a free area. And, the picture area that hasthe picture data that is already accumulated in the picture memory isnot deleted.

The head part of FIG. 14B is an example of the case where the picturememory 203 does not have enough free area to accumulate the decodedpicture C. In such example as described above, P4, P5 and P7 of thepicture stream as shown in FIG. 3 are accumulated in the picture memory,and P8 is decoded (that is, C=P8) at present. In such case as describedabove, it is necessary to delete the picture area that has the picturedata accumulated in the picture memory 203 and to retain an area toaccumulate the newly decoded picture C. According to the firstembodiment, the picture to be decoded always has the picture size thatis the same as or smaller than the picture area that is an area made bydividing the picture memory. Thus, in the case where the picture memorydoes not have enough free area, the area for accumulating the decodedpicture can be retained by deleting one of the picture areas. Thepicture area where the picture which has the oldest display orderinformation is accumulated is selected as the picture area to bedeleted. In such example as described above, it is noticeable that thedisplay order information of P4 is the oldest. The deletable picturearea judgment unit 209 judges the picture area to be deleted by suchprocedures as described above. Then, the deletable picture area judgmentunit 209 outputs the judging result to the picture memory control unit206, and deletes the picture. In the example of the head part of FIG.14B, P4 is deleted by such procedures as described above, and thedecoded picture C is accumulated in the generated free area as shown inthe bottom part of FIG. 14B.

Although in the first embodiment as described above, when deleting thepicture area from the picture memory 203 for decoding, the picture areais sequentially deleted from the one whose accumulated picture has theoldest display order information, it is possible to perform the sameprocedures in the case of deleting the picture whose order to have beendecoded is the oldest, in stead of the display order information.

As recognized from such explanations as described above, according tothe first embodiment, it is not necessary to distinguish the followingtwo cases (a) and (b) because the picture memory 203 can be managed bythe same procedures: (a) the case where the picture size of the pictureC for decoding is changed, and (b) the case where the picture size ofthe picture C for decoding is not changed. In other words, in both caseswhere the picture size is changed or not changed when coding, by thesimple procedure that always deletes the picture area by one area,leaving as many pictures that may not be displayed as possible in thepicture memory, the picture memory can be managed without anycontradictions. Also, on the coding apparatus side, in addition to theconventional method, it is possible to perform the procedures of thefirst embodiment only by coding the information of the biggest picturesize included in the stream and writing the coded information in thecoded stream, thus the picture coding apparatus is easily implemented.However, as happened in the examples of FIG. 13A, FIG. 13B, FIG. 14A andFIG. 14B, in the case where the newly decoded picture (the backwardpicture from the picture C for decoding) has a small picture size, auseless area occurs in each picture area, the picture memory is notefficiently utilized. However, as it is possible to sequentiallyaccumulate a picture of any picture size, without being physicallydivided, on the picture memory, simplifying the access to the picturememory, thereby it is possible to perform decoding without lowering thespeed. Also, in the regular picture coding and decoding, even if thepicture size is changed, the structure that performs coding and decodingwithout changing the reference relations of the pictures (for example,the number of the pictures to use for reference) is often used.Consequently, according to such method as explained in the firstembodiment, it is possible to perform decoding by the efficientprocedures.

Second Embodiment

According to the first embodiment, based on the information of thebiggest picture size that may be included in the stream, the picturememory 203 on the decoding side is divided; and with the picture area asa unit, the picture memory 203 is managed. On the other hand, accordingto the second embodiment, based on the information of the smallestpicture size that may be included in the stream, the picture memory 203is managed.

The coding method of a moving picture according to the second embodimentwill be explained using a block diagram as shown in FIG. 8. Thestructure of the picture coding apparatus is similar to that of thepicture coding apparatus 800 as explained in the first embodiment. Here,the explanation of the same structure elements as the ones alreadyexplained will be omitted.

Preceding the coding of the stream for coding, the picture memorydividing information coding unit 801 codes the information to specifythe method to divide the picture memory on the decoding side. Accordingto the second embodiment, as the information to specify the method todivide the picture memory 203 on the decoding side, the information ofthe smallest picture size that may be included in the stream is coded.FIG. 15 is a diagram showing a data structure of a coded streamgenerated by the picture coding apparatus 800 of the second embodiment.FIG. 15A is a diagram showing an example of a coded stream in the casewhere the information of the smallest picture size that may be includedin the stream is coded. The picture memory dividing information codingunit 801 writes the information of the smallest picture size that may beincluded in the stream as min_size of the picture common informationarea of the coded stream as shown in FIG. 15A. Here, as the smallestpicture size, for example, the smallest picture size which the picturecoding apparatus can use for coding can be used; otherwise, a user canselect and determine one of the suggested plural picture sizes whenperforming the coding.

In the case where the picture for coding is a picture capable ofchanging the size of an IDR picture or an equivalent picture thereto, aswell as the conventional technology, the picture memory control unit 109inputs a signal indicating the reset method of the picture memory intothe coded stream generation unit 103. According to the presentembodiment, the reset method of the picture memory specified in suchcase as described above is arbitrary. For example, in the case of theoperation of the picture memory on the decoding side, depending on theexistence of the picture that is not displayed in the picture memory,“reset_flag is set 0”, that is, all the pictures are made “non-referencepictures”; and the picture data itself is not deleted. Otherwise, it ispossible to arbitrarily select either of the two: (a) “reset_flag is set0”; or (b) “reset_flag is set 1”, that is, that the data itself of allthe pictures is deleted and all the area of the picture memory isemptied. The coded stream generation unit 103 codes the inputted signal,and generates the coded stream that has reset_flag in the slice headeras shown in FIG. 15A.

Also, in stead of writing the smallest picture size, as min_size, thatmay be included in the stream, there is a method to directly write thesize of each of the divided picture areas, as one_mem_size, afterdividing the picture memory 203 for decoding. FIG. 15B is a diagramshowing an example of a coded stream in the case where the informationof the size of the picture area that can be acquired by dividing thepicture memory 203 is coded. For example, as FIG. 15B shows, it ispossible to write the size of the divided picture area directly asone_mem_size in the picture common information. As such one_mem_size,for example, (i) the smallest picture size included in the stream can beused as it is, (ii) the greatest common divisor of all the picture sizesincluded in the stream can be used, and (iii) the size of a group ofareas where the most efficient access is possible in the operation ofthe picture memory on the decoding side can be used.

When determining one_mem_size, in order to accumulate one macroblock inthe same area, or in order to accumulate all the macroblocks of onehorizontal stream of a picture in the same area, it is possible toefficiently manage the picture memory by rounding up the value ofone_mem_size. Also, such round-up as described above can be done on thepicture decoding apparatus side. For example, there is a case where thepictures of three kinds of picture sizes are coded in one stream asshown in FIG. 16A. FIG. 16A is a diagram showing an example of threekinds of picture sizes that are coded in one stream. In FIG. 16A thesize of each picture is expressed as to how many macroblocks (forexample, 16×16 pixels) the vertical and horizontal lengths correspondto. In FIG. 16A the picture A is formed with such macroblocks asvertical 6 and horizontal 7; the picture B is formed with suchmacroblocks as vertical 8 and horizontal 10; and the picture C is formedwith such macroblocks as vertical 12 and horizontal 12. FIG. 16B is adiagram showing an example of the method to divide the picture area toefficiently use the picture memory 203 in the case where one stream hasthe pictures of three sizes as shown in FIG. 16A. In such case asdescribed above, if the size of the area where the data of 48macroblocks can be accumulated is written as one_mem_size, as FIG. 16Bshows, it is possible to accumulate the data of one picture asfollowing: the picture A in one picture area; the picture B in twopicture areas; and the picture C in three picture areas. Here, it isnoticeable that the macroblocks of one horizontal stream in each pictureare distributed to one picture area without being intercepted in themiddle. Although all the pictures are equally divided in the example ofFIG. 16B, it is possible to treat the unequally divided pictures withoutany problems.

Also, in stead of writing the size information of one picture area whichis made by dividing the picture memory on the decoding side, asone_mem_size, it is possible to write the number of the divided pictureareas as area_num in the picture common information area of the codedstream. FIG. 15C is a diagram showing an example of a coded stream inthe case where the number of the picture areas that can be acquired bydividing the picture memory 203 is coded. For example, as FIG. 15Cshows, the number of the divided picture areas can be directly written,as area_num, in the picture common information. In such case asdescribed above, the size of the picture memory of the picture decodingapparatus is uniform according to the standard, the value of area_num iseasily determined by the equation 4 as shown below. Here, thecalculation result of the division in the equation is calculated as aninteger number by the round-down procedure.(area_num)=(the size of the picture memory of the picture decodingapparatus 1100)/(one_mem_size)  (Equation 4)

Also, it is possible to treat such picture size and picture memory sizeas described above in the same manner, no matter which of the followingsthey may be: the whole number of pixels, the whole number ofmacroblocks, and the whole number of bits.

FIG. 17A, FIG. 17B and FIG. 17C are diagrams showing an example of acoded stream in the case where additional information is written in thearea of the picture common information of each coded stream as shown inFIG. 15A, FIG. 15B and FIG. 15C. Also, each coded stream as shown inFIG. 15A, FIG. 15B and FIG. 15C can be written as shown in FIG. 17A,FIG. 17B and FIG. 17C. In other words, in addition to min_size,one_mem_size and area_num, cnf_flag can be written in the picture commoninformation area of the coded stream. cnf_flag indicates whether or notall the pictures included in the stream can be displayed without anyproblems, even in the case where the picture memory on the decoding sideis divided and controlled according to min_size, one_mem_size andarea_num. For example, in the case where the value of cnf_flag is 1, itindicates that all the pictures in the stream can be displayed withoutany problems. In the case where the value of cnf_flag is 0, it indicatesthe possibility that the pictures that are not displayed yet may not bedisplayed because the data is already lost, due to the procedures of thesecond embodiment.

Although in FIG. 15A, FIG. 15B and FIG. 15C and FIG. 17A, FIG. 17B andFIG. 17C min_size, one_mem_size, area_num and cnf_flag are written inthe picture common information area, it is also possible to writemin_size, one_mem_size, area_num and cnf_flag in the data area where thedata that is not essential, but supports the decoding procedures iscollected. Also, it is possible to write min_size, one_mem_size,area_num and cnf_flag in the sequence common information area that canbe referred to by all the sequences. Moreover, it is possible to writemin_size, one_mem_size, area_num and cnf_flag in the slice header.

Next, the decoding method of the coded stream coded by such codingmethod as described above will be explained by using a block diagram ofthe picture decoding apparatus 1100 as shown in FIG. 11, a flow chart ofthe decoding procedure as shown in FIG. 12, and a schematic diagram ofthe picture memory as shown in FIG. 18A, FIG. 18B, FIG. 19A and FIG.19B.

As well as the first embodiment, according to the second embodiment, thepicture memory is not managed per picture; however, the so-calledpicture area that is made by dividing the picture memory into one orplural areas is used as a unit for holding and deleting a picture.Preceding the decoding of the coded stream, the picture memory dividingmethod specification unit 210 analyzes the dividing method of thepicture memory from a signal of the coded stream, and directs thepicture memory control unit 206. According to the second embodiment, asthe information for specifying the dividing method of the picturememory, the information of the smallest picture size that may beincluded in the stream is written in the picture common information areaas min_size that is included in the stream of FIG. 15A. The picturememory dividing method specification unit 210 analyzes the signal, anddetermines the dividing method of the picture memory. The number of theareas divided is calculated by the equation 5 as shown below. Also, thecalculation result of the division in the equation is calculated as aninteger number by the round down procedure.(The number of the picture areas)=(The size of the picture memory of thepicture decoding apparatus 1100)/(min_size)  (Equation 5)

Also, in the case where the coded stream of FIG. 15B is used in stead ofthe coded stream of FIG. 15A, the number of the picture areas iscalculated by the equation 6 as shown below.(The number of the picture areas)=(The size of the picture memory of thepicture decoding apparatus 1100)/(one_mem_size)  (Equation 6)

Also, in the case where the coded stream of FIG. 15C is used, in steadof the coded stream of FIG. 15A, the number of the picture areas can becalculated by the equation 7 as shown below.(The number of the picture areas)=(area_num)  (Equation 7)

Here, the picture size and the picture memory size as described abovecan be treated in the same manner, no matter which of the three they maybe: the whole number of pixels, the whole number of macroblocks and thewhole number of bits.

The picture size change judgment unit 207 compares the picture size ofthe picture to be decoded and the picture size of the previously (orjust) coded picture, and judges whether or not the picture size has beenchanged.

reset_flag is coded in the slice header of the coded stream, as shown inFIG. 15A, FIG. 15B and FIG. 15C. The coded stream analysis unit 201analyzes the value of the reset_flag, and outputs the information of theanalysis result to the picture memory control unit 206. According to thesecond embodiment, even if the picture size was changed, there is apossibility that the value of reset_flag is selected as either 0 or 1.Thus, according to the value of reset_flag, the management method of thepicture memory is divided as shown in the flow chart of FIG. 12. Inparticular, in the case where reset_flag is 0, there is a possibilitythat the picture before the change of the picture size remains in thepicture memory; therefore, it is necessary to judge which picture areashould be deleted in order to accumulate the picture after the change ofthe picture size. The deletable picture area judgment unit 209 makessuch judgment as described above; and the judging result is outputted tothe picture memory control unit 206. Consequently, the picture memory ismanaged.

Next, the specific operations of the picture memory control unit 206,the picture size change judgment unit 207 and the deletable picture areajudgment unit 209 in the decoding procedure will be explained using theflow chart of FIG. 12. Although in the flow chart in FIG. 12, the changeof the picture size and the value of reset_flag are judged after thecurrent picture is decoded, such order is arbitrary and not limited tothe order as shown in FIG. 12.

First, the flow of the decoding procedure will be explained in the casewhere the picture size of the picture to be decoded is changed (S1204)and reset_flag is 1 (S1205). The value of reset_flag as 1 indicates thatall the picture data in the picture memory is deleted and all the areais emptied. In other words, if such signal as described above isinputted into the picture memory control unit 206, it is possible thatthe picture memory control unit 206 deletes all the area of the picturememory (S1206), the decoded picture after the change of the picture sizeis saved in the arbitrary area in the picture memory (S1207).

Next, the flow of the decoding procedure will be explained in the casewhere the picture size of the picture to be decoded is changed (S1204),and reset_flag is 0 (S1205). The value of reset_flag as 0 indicates thatall the picture data in the picture memory is made “non-reference”pictures, but the picture data itself is not deleted and leftaccumulated. The deletable picture area judgment unit 209 checks thefree area of the picture memory. In the case where there is not enoughfree area, the deletable picture area judgment unit 209 compares (a) thesize of the picture after the picture size change and (b) the size ofone picture area which is made by dividing the picture memory. Then, thedeletable picture area judgment unit 209 determines the dividing methodof the decoded picture, and judges the number of the picture areas to bedeleted and the picture areas to be deleted (S1208). The picture that isaccumulated in the picture memory 203 has the display order information,in addition to the picture data. The deletable picture area judgmentunit 209 checks the display order information. Then, the deletablepicture area judgment unit 209 specifies the picture area where thepicture that has the oldest display order information is accumulated,and transmits the direction to sequentially delete the specified pictureareas to the picture memory control unit 206. Consequently, thespecified picture areas are deleted, and the decoded picture isaccumulated in the generated free area (S1210).

FIG. 18A and FIG. 18B are diagrams showing a situation where the newlydecoded picture is accumulated in the picture memory 203, following theorder of the steps S1204-S1210 of the flow chart as shown in FIG. 12.

FIG. 18A is an example of the case where there is enough free area toaccumulate the decoded picture C in the picture memory 203. In suchexample as described above, P2, P6, P4 and P5 of the picture stream inFIG. 3 are accumulated in the picture memory, and P7 is decoded atpresent (that is, C=P7). P2, P6, P4 and P5 are the pictures that havethe same picture size as the divided picture area. On the other hand,the newly decoded picture C has the picture size that is bigger than onepicture area. The example of FIG. 18A indicates that the picture C isdivided into two and accumulated. In such case as described above, it ispossible to accumulate the decoded picture C in the free area as it is,using the two picture areas as shown in the bottom part of FIG. 18A. Inother words, the picture area where the picture data is alreadyaccumulated is not deleted.

The head part of FIG. 18B is an example of the case where there is notenough free area to accumulate the decoded picture C in the picturememory. In such example as described above, P3, P1, P2, P6, P4 and P5 ofthe picture stream as shown in FIG. 3 are accumulated in the picturememory, and P7 is decoded at present (that is, C=P7). In such case asdescribed above, it is necessary to delete the picture area that has thepicture data before the change of the picture size and to retain an areato accumulate the decoded picture C. In such case as described above,the picture C, after the change of the picture size, has the picturesize that is bigger than one picture area. Thus, it should be noted thatthe picture C is divided into two and accumulated using two pictureareas. The picture area where the accumulated picture has the oldestdisplay order information is selected as the picture area to be deleted.In such example as described above, it should be noted that the displayorder information of P1 and P2 is the oldest in such order. Thedeletable picture area judgment unit 209 judges the picture area to bedeleted by such procedures as described above. Then, the deletablepicture area judgment unit 209 outputs the judging result to the picturememory control unit 206, and deletes the picture area. In the example ofthe head part of FIG. 18B, P1 and P2 are deleted by such procedures asdescribed above; and the decoded picture C as shown in the bottom partof FIG. 18B is accumulated in the newly generated free area.

Finally, the flow of the decoding procedure will be explained in thecase where the picture size of the picture to be decoded is not changedwhen decoding, but is already changed before decoding, or in the casewhere the picture size of the picture to be decoded is not changed atall (S1204). According to the second embodiment, regardless of theexistence of the picture whose size is different from the picture to bedecoded in the picture memory 203, it is possible to manage the picturememory by the same procedure as described above. The deletable picturearea judgment unit 209 checks the free area of the picture memory. Inthe case where there is not enough free area, the deletable picture areajudgment unit 209 compares (a) the size of the picture after the picturesize change and (b) the size of one picture area which is made bydividing the picture memory. Then, the deletable picture area judgmentunit 209 determines the dividing method of the decoded picture, andjudges the number of the picture areas to be deleted and the pictureareas to be deleted. The picture that is accumulated in the picturememory has the display order information, in addition to the picturedata. The deletable picture area judgment unit 209 checks the displayorder information. Then, the deletable picture area judgment unit 209specifies the picture area where the picture whose display orderinformation is the oldest is accumulated, and transmits the direction tosequentially delete the specified picture area to the picture memorycontrol unit 206.

FIG. 19A and FIG. 19B are diagrams showing a situation where the newlydecoded picture is accumulated in the picture memory 203 following thesteps S1204-S1213 of the flow chart as shown in FIG. 12.

FIG. 19A is an example of the case where the picture memory has enoughfree area to accumulate the decoded picture C. Such example as describedabove shows the situation where P4, P5 and P7 of the picture stream areaccumulated in the picture memory and P8 is decoded (that is, C=P8) atpresent. P4 and P5 have the same picture size as the picture area. Onthe other hand, the newly decoded picture C, after the change of thepicture size, has the bigger picture size than one picture area. Theexample of FIG. 19A indicates that the picture C is divided into two andaccumulated. In such case as described above, as shown in the bottompart of FIG. 19A, it is possible to accumulate the decoded picture C asit is in the free area, using two picture areas. In other words, thepicture area that has the picture data that is already accumulated inthe picture memory is not deleted.

The head part of FIG. 19B is an example of the case where the picturememory 203 does not have enough free area to accumulate the decodedpicture C. In such example as described above, P2, P6, P4, P5 and P7 ofthe picture stream as shown in FIG. 3 are accumulated in the picturememory, and P8 is decoded (that is, C=P8) at present. In such case asdescribed above, it is necessary to delete the picture area that has thepicture data and to retain an area to accumulate the newly decodedpicture C. In such case as described above, the decoded picture C, afterthe change of the picture size, has the bigger picture size than onepicture area. The example of FIG. 19B indicates that the picture C isdivided into two and accumulated. Also, the picture area whose picturehas the oldest display order information is selected as the picture areato be deleted. In such example as described above, it is noticeable thatthe display order information of P2 and P4 are the oldest in such order.The deletable picture area judgment unit 209 judges the picture area tobe deleted by such procedures as described above. Then, the deletablepicture area judgment unit 209 outputs the judging result to the picturememory control unit 206, and deletes the picture. In the example of thehead part of FIG. 19B, P2 and P4 are deleted by such procedures ofjudging the deletable picture areas as described above, and the newlydecoded picture C is accumulated in the generated free area as shown inthe bottom part of FIG. 19B.

Although in the second embodiment as described above, when deleting thepicture area from the picture memory 203 for decoding, the picture areais sequentially deleted from the one whose picture has the oldestdisplay order information, it is possible to perform the same proceduresin the case of deleting the picture whose order to be decoded is theoldest, in stead of the display order information.

As recognized from such explanations as described above, according tothe second embodiment, it is not necessary to distinguish the followingtwo cases (a) and (b) because the picture memory 203 can be managed bythe same procedures: (a) the case where the picture size of the pictureC is changed, and (b) the case where the picture size of the picture Cis not changed. In other words, in both cases where the picture size ischanged or not changed when coding, by the simple processing method thatalways deletes only the necessary picture area(s) to be deleted byjudging how many areas are necessary for the decoded pictures, leavingas many pictures that may not be displayed as possible in the picturememory, the picture memory can be managed without any contradictions.Also, on the coding side, in addition to the conventional method, it ispossible to perform the procedures of the second embodiment only by (i)coding (a) the information of the smallest picture size included in thestream or (b) the size of one picture area which is made by dividing thepicture memory, on the decoding side, and (ii) writing the codedinformation in the coded stream, thus the picture coding apparatus iseasily implemented. Also, it is possible to arbitrarily set the dividingsize of the decoded picture in consideration with the efficiency of thememory access. Consequently, decoding can be performed without loweringthe speed of the memory access.

Third Embodiment

According to the first embodiment, the information to specify thedividing method of the picture memory is generated in the picture codingapparatus, and written in the coded stream. On the other hand, accordingto the third embodiment, the dividing method of the picture memory isanalyzed and decided in the picture decoding apparatus. Thus, the codingmethod of the moving picture according to the third embodiment is thesame as the conventional coding method as explained in FIG. 1; and theexplanation will be omitted here.

FIG. 20 is a block diagram showing a structure of the picture decodingapparatus 2000 according to the third embodiment. The decoding method ofthe moving picture according to the third embodiment will be explained,using a block diagram as shown in FIG. 20, as following. The structureof the decoding procedure is similar to the method according to thefirst embodiment as explained using FIG. 11. The difference from themethod according to the first embodiment is that the address divisionunit 210A is provided, in place of the picture memory dividing methodspecification unit 210. Here, the same procedures as the firstembodiment will be omitted.

The picture size change judgment unit 207 compares (a) the picture sizeof the picture to be decoded and (b) the picture size of the picturewhich has been decoded previously (or just before), and judges whetheror not the picture size has been changed. When the picture size changejudgment unit 207 judges that the picture size has been changed, itnotifies the picture size after the change to the address division unit210A via the coded stream analysis unit 201. The address division unit210A divides the address space of the picture memory 203 so that thepicture size, after the picture size change, which is notified by thepicture size change judgment unit 207 becomes the picture size of eachpicture area. And, the address division unit 210A directs the picturememory control unit 206 to manage the picture memory 203 using theaddress after the division as a unit.

The number of the areas which are divided as described above iscalculated by the equation 8 as shown below. Here, the calculationresult of the division in the equation is calculated as an integernumber by the round down procedure.(the number of the picture areas)=(the size of the picture memory 203 ofthe picture decoding apparatus 2000)/(the picture size after the picturesize change)  (Equation 8)

Also, the picture size and the picture memory size as described abovecan have any of the followings as a unit: the whole number of pixels,the whole number of macroblocks and the whole number of bits; and all ofthem can be treated in the same manner.

Next, the specific operations of the decoding procedure of the picturememory control unit 206, the picture size change judgment unit 207, thedeletable picture area judgment unit 209 and the address division unit210A will be explained using FIG. 21. FIG. 21 is a flow chart showingthe order of the decoding procedure of the picture decoding apparatus2000 as shown in FIG. 20.

First, when the picture size of the current picture to be decoded ischanged (S2301), the address division unit 210A acquires the picturesize of the current picture to be decoded, that is, the picture size,after the picture size change, from the picture size change judgmentunit 207, via the coded stream analysis unit 201 (S2302). The addressdivision unit 210A divides the address space of the picture memory 203with the acquired picture size. And, the address division unit 210Adirects the picture memory control unit 206 to manage the picture memory203 using the divided picture area as a unit (S2303). In the case wherethe picture size of the picture to be decoded is not changed (S2301),the procedures of the step S2302 and the step S2303 are not performed.

Next, the deletable picture area judgment unit 209 checks the free areaof the picture area 203 (S2304). In the case where none of the dividedpicture areas exists as a serial free area, the deletable picture areajudgment unit 209 sequentially deletes the picture of the picture memory203 from the one whose display order is the oldest, until at least oneof the picture areas becomes a free area (S2305). Specifically, thepicture which is accumulated in the picture memory 203 has theinformation that indicates the display order of the picture, in additionto the picture data. The deletable picture area judgment unit 209 checkssuch display order information as described above, and transmits thedirection to sequentially delete the picture from the picture whosedisplay order information is the oldest to the picture memory controlunit 206. The picture memory control unit 206 repeats the procedure todelete the picture specified by the deletable picture area judgment unit209 until one of the divided picture areas becomes a free area, andaccumulates the newly decoded picture in the open picture area (S2306)when one of the divided picture areas becomes a free area. In the casewhere one of the divided picture areas is already a free area (S2304),the newly decoded picture is accumulated as it is in such picture areaas described above (S2306).

FIG. 22A and FIG. 22B are diagrams showing the situation where the newlydecoded picture is accumulated in the picture memory 203, according tothe order of the flow chart as shown in FIG. 21. FIG. 22A and FIG. 22Bshow the case where the picture size of the backward picture from thepicture C for decoding is changed to a larger size. FIG. 22A shows anexample of the case where the picture memory 203 has enough free area toaccumulate the newly decoded picture C. FIG. 22A shows the situationwhere P2, P4, P5 and P6 of the picture stream in FIG. 3 are accumulatedin the picture memory, and P7 is decoded at present (that is, C=P7). InFIG. 22A the picture size is changed from the picture P7 which is thepicture C for decoding. As the bottom part of FIG. 22A shows, theaddress of the picture memory 203 is divided by the picture size of P7.In such case as described above, as the bottom part of FIG. 22A shows,it is possible to accumulate the newly decoded picture C as it is in thefree area. The picture area that has the picture data that is alreadyaccumulated in the picture memory is not deleted.

FIG. 22B shows an example of the case where the picture memory 203 doesnot have enough free area to accumulate the newly decoded picture C.Such example as described above shows the situation where P1, P2, P3,P4, P5 and P6 of the picture stream in FIG. 3 are accumulated in thepicture memory 203, and P7 is decoded at present (that is, C=P7). Inthis case, it is necessary to delete the area that has the picture dataaccumulated in the picture memory 203, and to retain the area toaccumulate the newly decoded picture C. According to the thirdembodiment, the picture C for decoding always has the same picture sizeas the picture area that is made by dividing the picture memory. Thus,in the case where the picture memory 203 does not have a free area, bydeleting the picture until one of the picture areas in the picturememory completely becomes a free area, the area for accumulating thenewly decoded picture can be retained. The picture to be deleted issequentially selected from the one whose display order information isthe oldest among the accumulated pictures. In such example as describedabove, it should be noted that the display order information of P1 isthe oldest. However, if only P1 is deleted, none of the picture areasthat are divided, adjusting to the picture size of the picture C fordecoding, can be a free area. At the stage where P2 and P3 are deletedafter P1, according to the display order information, the second picturearea from the head address of the picture memory 203 becomes one freearea. The deletable picture area judgment unit 209 judges the picture tobe deleted by such procedure as described above, and outputs thejudgment result to the picture memory control unit 206. The picturememory control unit 206 deletes the picture according to the judgmentresult of the deletable picture area judgment unit 209. In the exampleof the head part of FIG. 22B, P1, P2 and P3 are deleted by suchprocedure as described above; and as the bottom part of FIG. 22B shows,the newly decoded picture C is accumulated in the generated free area.

FIG. 23A and FIG. 23B are diagrams showing the situation where the newlydecoded picture is accumulated in the picture memory 203, according tothe order of the flow chart as shown in FIG. 21. FIG. 23A and FIG. 23Bshow the case where the present picture size of the backward picturefrom the picture C for decoding is changed to a smaller size. FIG. 23Ashows an example of the case where the picture memory 203 has enoughfree area to accumulate the newly decoded picture C. In this example,P1, P2 and P3 of the picture stream as shown in FIG. 3 are accumulatedin the picture memory; and P4 is decoded at present (that is, C=P4). Insuch example as described above, the picture size of P4, which is thepicture C for decoding, is changed; and as the bottom part of FIG. 23Ashows, the address of the picture memory 203 is divided by the picturesize of P4. In such case as described above, as the bottom part of FIG.23A shows, it is possible to accumulate the newly decoded picture C in apicture area of the free area without any discrepancies. And, thepicture area that has the picture data that is already accumulated inthe picture memory is not deleted.

FIG. 23B shows an example of the case where the picture memory 203 doesnot have enough free area to accumulate the newly decoded picture C. Inthis example, P1, P2, P3 and P4 of the picture stream as shown in FIG. 3are accumulated in the picture memory 203; and P5 is decoded at present(that is, C=P5). In such case as described above, it is necessary todelete the area where the picture data is already accumulated and toretain an area to accumulate the newly decoded picture C in the picturememory 203. According to the third embodiment, the picture C fordecoding always has the same picture size as the picture area that ismade by dividing the picture memory. Thus, in the case where the picturememory 203 does not have a free area, by deleting the picture until oneof the picture areas in the picture memory completely becomes a freearea, the area for accumulating the newly decoded picture can beretained. The picture to be deleted is sequentially selected from theone whose display order information is the oldest among the accumulatedpictures. In such example as described above, it should be noted thatthe display order information of P1 is the oldest. By deleting P1, onepicture area becomes a free area. The deletable picture area judgmentunit 209 judges the picture to be deleted by such procedure as describedabove, and outputs the judgment result to the picture memory controlunit 206. The picture memory control unit 206 deletes the pictureaccording to the judgment result of the deletable picture area judgmentunit 209. In the example of the head part of FIG. 23B, P1 is deleted bysuch procedure as described above; and as the bottom part of FIG. 23Bshows, the newly decoded picture C is accumulated in the generated freearea.

Although according to the third embodiment, when the picture area isdeleted from the picture memory 203, the picture area is sequentiallydeleted from the one where the accumulated picture has the oldestdisplay information, the same procedures can be performed in the casewhere the picture area is deleted from the one where the accumulatedpicture has the oldest turn to be decoded, in stead of the display orderinformation.

As recognized from such explanation as described above, according to thethird embodiment, it is not necessary to distinguish the following twocases (a) and (b) because the picture memory 203 can be managed by thesame procedures: (a) the case where the picture size of the picture Cfor decoding is changed, and (b) the case where the picture size of thepicture C for decoding is not changed. In other words, in both caseswhere the picture size is changed or not changed when coding, by thesimple procedure that always deletes the stored picture until onepicture area completely becomes a free area, leaving as many picturesthat may not be displayed as possible in the picture memory, the picturememory can be managed without any contradictions. Also, each size of thenewly divided picture areas is the same as the pictures to be newlydecoded (the backward pictures of the picture C for decoding); thus, inboth cases where the picture size is changed to a larger size and asmaller size, the useless area that is not used does not occur in eitherof the picture areas. Therefore, in the picture decoding apparatus 2000of the third embodiment, the memory area of the picture memory 203 isefficiently utilized. In addition, it is possible to sequentiallyaccumulate a picture of any picture size, without physically divided, onthe picture memory, simplifying the access to the picture memory,thereby it is possible to perform decoding without lowering the speed.Also, in the regular picture coding and decoding, even if the picturesize is changed, the structure that performs coding and decoding withoutchanging the reference relations of the pictures (for example, thenumber of the pictures to use for reference) is often used.Consequently, according to such method as explained in the thirdembodiment, it is possible to perform decoding by the efficientprocedures.

However, according to the third embodiment, the picture data is notaccumulated from the head of the free area, but the head of the picturearea of the free area; therefore, as FIG. 22B shows, there is apossibility that the area that is bigger than the space to accumulateone picture C for decoding is deleted. Thus, in order to accumulate thenewly decoded picture C, when the picture is sequentially deleted fromthe one whose display order is the oldest until one picture area becomesa free area, especially in the case where the picture whose displayorder information is old is not accumulated in the display order in theserial area within the picture memory 203, there is a possibility thatthe “display-waiting picture” may be deleted as well.

According to the third embodiment, the picture is deleted from the onewhose display order information is the oldest; and when one picture areabecomes a serial free area, the newly decoded picture C is accumulatedin the picture area. However, the present invention is not limited tosuch method as described above; and the picture C for decoding can beaccumulated in a different manner. In other words, regardless of thedisplay order information of the picture, the already accumulatedpicture is sequentially deleted from the head address until one picturearea of the head of the picture memory 203 becomes a serial free area.When a head picture area becomes a serial free area, the newly decodedpicture C is accumulated in the picture area. Next, in the case wherethe decoded picture C is accumulated, the pictures are deleted in theorder of the address from the next address of the picture that isdeleted last, until the second picture area from the head of the picturememory 203 becomes a serial free area. FIG. 24 is a flow chart showingthe order of the other decoding procedures in the picture decodingapparatus 2000 as shown in FIG. 20.

In FIG. 24 the procedures of the step S2601—the step S2603 are the sameas the procedures of the step S2301—the step S2303 as shown in FIG. 21.By such procedures as described above, (i) the picture size of thepicture to be decoded is changed; (ii) the address space of the picturememory 203 is divided by the picture size after the picture size change;and (iii) the picture memory 203 is managed using the divided picturearea as a unit.

Next, the deletable picture area judgment unit 209 checks the free areaof the picture memory 203. If the picture memory 203 has enough freearea including the picture area to accumulate the newly decoded picture,the deletable picture area judgment unit 209 accumulates the newlydecoded picture as it is in the open picture area (S2606). If thepicture area to accumulate the newly decoded picture does not exist as afree area (S2604), the deletable picture area judgment unit 209 makesthe next picture area to the picture area where the decoded picture isaccumulated last a free area, sequentially from the head of the picturememory 203 (S2605). And, the deletable picture area judgment unit 209accumulates the newly decoded picture in the open picture area (S2606).FIG. 25 is a flow chart showing the more specific order of the procedureof the step S2605 as shown in FIG. 24. More specifically, the picturememory control unit 206 is equipped with a pointer that indicates apredetermined address location within the picture memory 203 that isused only when the picture memory 203 does not have enough free area toaccumulate a new picture. Here, the predetermined address location is,for example, the head address of each picture area. Also, the initialvalue of the pointer is set as “0” which indicates the head address ofthe head picture area of the picture memory 203. In addition, thepointer=(N−1) indicates the head address of the Nth picture area fromthe head of the picture memory 203 (N is a natural number).

In the case where it is judged that the picture area to accumulate thenewly decoded picture does not exist as a free area (S2604), thedeletable picture area judgment unit 209 has the picture memory controlunit 206 delete the picture in the picture area that is indicated by thepointer, in the address order (S2501-S2503). For example, when thepointer=0, the picture in the picture memory 203 is sequentially deletedfrom the head of the picture memory until the head picture area of thepicture memory 203 becomes a free area. When the head picture area inthe picture memory 203 becomes a free area (S2502), the picture memorycontrol unit 206 increments the pointer as “1” (S2504), and has thepointer point at the head address of the next picture area.

Here, the picture memory control unit 206 checks whether or not thevalue of the pointer is bigger than the number of all the picture areas.When the value of the pointer becomes bigger than the number of all thepicture areas (S2505), the value of the pointer is returned to “0”; andthe head address of the head picture area in the picture memory 203 ispointed at. In the case where the value of the pointer becomes biggerthan the number of all the picture areas, it indicates that newlydecoded pictures are accumulated in all the picture areas in the picturememory 203.

According to such procedures as described above, in the case where thepicture memory 203 does not have an open picture area to accumulate anewly decoded picture, the picture data is sequentially deleted from thehead of the picture memory; and a newly decoded picture is accumulatedin the open picture area.

FIG. 26A and FIG. 26B are diagrams showing the situation where the newlydecoded picture is accumulated in the picture memory 203, according tothe order of the flow chart as shown in FIG. 24 and FIG. 25. FIG. 26Aand FIG. 26B show the case where the picture size of the picture afterthe present picture C for decoding is changed to a larger size. FIG. 26Ashows an example of the case where the picture memory 203 has enoughfree area to accumulate the newly decoded picture C. FIG. 26A shows thesituation where P2, P4, P5 and P6 of the picture stream as shown in FIG.3 are accumulated in the picture memory, and P7 is decoded at present(that is, C=P7). In FIG. 26A the picture size is changed from thepicture P7 which is the picture C for decoding. As the bottom part ofFIG. 26A shows, the address of the picture memory 203 is divided by thepicture size of P7. In such case as described above, as the obliquelined part 2 of the head part of FIG. 26A shows, because the head partof the picture memory 203 has enough free area, it is possible toaccumulate the newly decoded picture C as it is in the free area. Thus,the picture area that has the picture data that is already accumulatedin the picture memory is not deleted.

FIG. 26B shows an example of the case where the picture memory 203 doesnot have enough free area to accumulate the newly decoded picture C.Such example as described above shows the situation where P1, P2, P3,P4, P5 and P6 of the picture stream as shown in FIG. 3 are accumulatedin the picture memory 203, and P7 is decoded at present (that is, C=P7).In this case, it is necessary to delete the area that has the picturedata accumulated in the picture memory 203, and to retain the area toaccumulate the newly decoded picture C. According to the thirdembodiment, the picture C for decoding always has the same picture sizeas the picture area that is made by dividing the picture memory. Thus,in the case where the picture memory 203 does not have a free area, bydeleting the picture until one of the picture areas in the picturememory completely becomes a free area, the area for accumulating thenewly decoded picture can be retained. The picture to be deleted issequentially selected in the address order from the head of the picturememory 203. In such example as described above, at the stage where P4and P1 are deleted in such order, the first picture area from the headof the picture memory 203 becomes one free area. The deletable picturearea judgment unit 209 judges the picture to be deleted by suchprocedure as described above, and outputs the judgment result to thepicture memory control unit 206. The picture memory control unit 206deletes the picture according to the judgment result of the deletablepicture area judgment unit 209. In the example of the head part of FIG.26B, P4 and P1 are deleted by such procedure as described above; and asthe bottom part of FIG. 26B shows, the newly decoded picture C isaccumulated in the head of the picture area. Also, when the next pictureC for decoding is further decoded, with the picture memory 203 in thesituation as shown in the bottom part of FIG. 26B, the newly decodedpicture has to be accumulated. In such case as described above, P2 andP3 are deleted; and the newly decoded picture is accumulated in thesecond open picture area.

FIG. 27A and FIG. 27B are diagrams showing the situation where the newlydecoded picture is accumulated in the picture memory 203, according tothe order of the flow chart as shown in FIG. 24 and FIG. 25. FIG. 27Aand FIG. 27B show the case where the picture size of the picture afterthe present picture C for decoding is changed to a smaller size. FIG.27A shows an example of the case where the picture memory 203 has enoughfree area to accumulate the newly decoded picture C. In this example,P1, P2 and P3 of the picture stream as shown in FIG. 3 are accumulatedin the picture memory; and P4 is decoded at present (that is, C=P4). Insuch example as described above, the picture size of the pictures thatare backward of P4 which is a picture to be decoded, is changed; and asthe bottom part of FIG. 27A shows, the address of the picture memory 203is divided by the picture size of the picture C for decoding (P4). Insuch case as described above, as the bottom part of FIG. 27A shows, itis possible to accumulate the newly decoded picture C in a picture areaof the free area without any discrepancies. And, the picture area thathas the picture data that is already accumulated in the picture memoryis not deleted.

FIG. 27B shows an example of the case where the picture memory 203 doesnot have enough free area to accumulate the newly decoded picture C. Inthis example, P1, P2, P3 and P4 of the picture stream as shown in FIG. 3are accumulated in the picture memory 203; and P5 is decoded at present(that is, C=P5). In such case as described above, it is necessary to (i)delete the area where the picture data is already accumulated and (ii)retain an area to accumulate the newly decoded picture C in the picturememory 203. According to the third embodiment, the picture C fordecoding always has the same picture size as the picture area that ismade by dividing the picture memory. Thus, in the case where the picturememory 203 does not have a free area, by deleting the picture until oneof the picture areas in the picture memory completely becomes a freearea, the area for accumulating the newly decoded picture can beretained. The picture to be deleted is sequentially selected, in theorder of the address where the picture data is stored, from the head ofthe picture memory 203. In such example as described above, it should benoted that the address of P3 is the very head. By deleting P3, onepicture area of the head of the picture memory 203 becomes a free area.The deletable picture area judgment unit 209 judges the picture to bedeleted by such procedure as described above, and outputs the judgmentresult to the picture memory control unit 206. The picture memorycontrol unit 206 deletes the picture according to the judgment result ofthe deletable picture area judgment unit 209. In the example of the headpart of FIG. 27B, P3 is deleted by such procedure as described above;and as the bottom part of FIG. 27B shows, the newly decoded picture C isaccumulated in the head picture area of the picture memory 203. In suchsituation as described above, when the next picture C for decoding isfurther decoded, P2 is deleted from the situation as shown in the bottompart of FIG. 27B; and the newly decoded picture is stored in the secondpicture area from the head of the picture memory 203.

Although in the example explained using the diagrams of FIG. 24-FIG.27B, in the case where the picture memory 203 has enough free area toaccumulate the newly decoded picture, the newly decoded picture isaccumulated in one open picture area, the present invention is notlimited to such method as described above. For example, even in the casewhere the picture memory 203 has enough free area, the picture area canbe selected in the address order from the head of the picture memory203; and the picture data accumulated in the selected picture area canbe sequentially deleted per picture until the selected picture areabecomes a serial free area. As a result, the newly decoded picture isaccumulated in the open picture area. In other words, as shown in thehead part of FIG. 27A, even in the case where there is enough free areato accumulate the picture C for decoding, the picture P1 accumulated inthe head of the picture memory 203 can be deleted; and the picture C fordecoding can be accumulated in the head open picture area of the picturememory 203. Thus, in both cases where (a) the picture memory has enoughfree area to accumulate the newly decoded picture and (b) the picturememory does not have enough free area to accumulate the newly decodedpicture, the picture area can be selected in the address order from thehead of the picture memory in the same manner; and the newly decodedpicture can be accumulated in the selected picture area. Consequently,as there are a few branches of the judgment, it is possible to decreasethe throughput required for the deletable picture judgment unit 209 toretain a picture area.

As recognized from such explanations as described above, according tothe third embodiment, in both cases where the picture size is changed ornot changed when coding, by the simple procedure that always deletes thestored picture until one picture area completely becomes a free area,leaving as many pictures that may not be displayed as possible in thepicture memory, the picture memory can be managed without anycontradictions. Also, each size of the newly divided picture areas isthe same as the pictures to be newly decoded (the backward pictures ofthe picture C for decoding); thus, in both cases where the picture sizeis changed to a larger size and a smaller size, the useless area that isnot used does not occur in either of the picture areas. Therefore, inthe picture decoding apparatus 2000 of the third embodiment, the memoryarea of the picture memory 203 is efficiently utilized. In addition, itis possible to sequentially accumulate a picture of any picture size,without physically divided, on the picture memory, simplifying theaccess to the picture memory, thereby it is possible to perform decodingwithout lowering the speed. Also, in the regular picture coding anddecoding, even if the picture size is changed, the structure thatperforms coding and decoding without changing the reference relations ofthe pictures (for example, the number of the pictures to use forreference) is often used. Consequently, according to such method asexplained in the third embodiment, it is possible to perform decoding bythe efficient procedures.

However, according to the third embodiment, the picture is sequentiallydeleted from the head of the picture memory, ignoring the display order,until the free area is retained. Thus, especially when the picture thathas new display order information is stored in a location close to thehead of the picture memory, there is a possibility that even the“display-waiting picture” may be deleted.

Also, according to the third embodiment as described above, it isnecessary to delete one or plural pictures until one of the pictureareas of the picture memory completely becomes a free area, there is apossibility that even the picture that is not displayed yet may bedeleted. Here, it is possible to use cnf_flag as a flag to indicatewhether or not all the pictures included in the stream can be displayedwithout any problems. For example, when the value of cnf_flag is 1, itindicates that all the pictures can be displayed without any problems.When the value of cnf_flag is 0, it indicates the possibility that thepictures that are not displayed yet may not be displayed because thedata is already lost, due to the procedures of the third embodiment.Thus, by writing cnf_flag in the coded stream and checking the value ofthis signal, it is possible to judge whether or not the procedures ofthe third embodiment can be applied. For example, when the value ofcnf_flag is 0, it should be noted that if the procedures of the thirdembodiment are applied, there is a possibility that some of the picturescannot be displayed. Therefore, instead, it is possible to switch to theconventional method where the picture memory is managed per picture.Consequently, with a possibility that the memory area may be fragmented,all the pictures can be displayed without any problems.

Fourth Embodiment

The coding method of the moving picture according to the fourthembodiment of the present invention will be explained using a blockdiagram as shown in FIG. 28. FIG. 28 is a block diagram showing astructure of the picture coding apparatus 2700 according to the fourthembodiment. The structure of the coding procedure is similar to theconventional method as explained using FIG. 1. The difference from theconventional method is that the picture size change judgment unit 110 isadded. The explanation of the same procedures as the conventionaltechnology will be omitted here. FIG. 29 is a diagram showing a datastructure of the coded stream which is generated by the picture codingapparatus 2700 according to the fourth embodiment.

In the case where the picture for coding is a picture capable ofchanging the size of an IDR picture or an equivalent picture thereto,the picture size change judgment unit 110 compares (a) the picture sizeof the picture for coding and (b) the picture size of the picture thatis already coded, and judges whether or not the picture size has beenchanged.

In the case where the judgment result inputted by the picture sizechange judgment unit 110 indicates that the picture size has not beenchanged, the picture memory control unit 109 performs coding, using thesame method as the regular coding. On the other hand, in the case wherethe judgment result indicates that the picture size has been changed,the signal indicating that all the pictures in the picture memory aredeleted, regardless of the “reference pictures” or “non-referencepictures”, as a reset method of the picture memory when decoding, isoutputted to the coded stream generation unit 103. The coded streamgeneration unit 103 performs coding by setting reset_flag of the codedstream as shown in FIG. 29 as “1”, which indicates that all the picturesin the picture memory (regardless of the “reference pictures” and“non-reference pictures”) are deleted.

On the other hand, it is possible to perform decoding of the codedstream coded by such coding method as described above, using the samestructure as the block diagram used to explain the conventional methodas shown in FIG. 2. The explanation of the same procedures as theconventional technology will be omitted here.

reset_flag is coded in the slice header of the coded stream as shown inFIG. 29. The coded stream analysis unit 201 analyzes the value of thereset_flag, and outputs the information of the analysis result to thepicture memory control unit 206. In the case where the picture size hasbeen changed, the coded stream generated by such coding method asdescribed above directs that all the pictures in the picture memory bedeleted, regardless of the “reference pictures” and “non-referencepictures”. Thus, when decoding, without concerning whether or not thepicture size has been changed in the same manner as the conventionalmethod, the picture memory can be reset according to the value ofreset_flag. In the case where the picture size has been changed, (i) allthe area of the picture memory becomes empty, due to the direction ofthe reset_flag; (ii) the picture after the picture size change isconventionally decoded; (iii) the decoded picture is accumulated in thepicture memory; (iv) the pictures of different sizes do not exist in thepicture memory; thereby (v) it is possible to use an arbitrary area as alocation for accumulating the decoded picture.

As explained in the fourth embodiment, in the case where the picturesize has been changed when coding, it is possible to manage the picturememory without any contradictions by coding the flag that has the valueindicating that all the pictures in the picture memory are deleted andby performing the same procedures as the conventional method on thedecoding side.

Fifth Embodiment

Furthermore, the processing shown in each of the above embodiments 1-4can be carried out easily in an independent computer system by recordinga program for realizing the moving picture coding/decoding methodsdescribed in each of the above embodiments onto a recording medium suchas a flexible disk or the like.

FIG. 30A, FIG. 30B and FIG. 30C are an illustration of a recordingmedium for recording a program for realizing the moving picturecoding/decoding method described in the above embodiments in thecomputer system.

FIG. 30B shows a full appearance of a flexible disk, its structure atcross section and the flexible disk itself whereas FIG. 30A shows anexample of a physical format of the flexible disk as a main body of arecording medium. A flexible disk FD is contained in a case F with aplurality of tracks Tr formed concentrically from the periphery to theinside on the surface of the disk, and each track is divided into 16sectors Se in the angular direction. Thus, the moving picture codingmethod and the moving picture decoding method as the program arerecorded in an area assigned for it on the flexible disk FD.

FIG. 30C shows a configuration for recording and regenerating theprogram on the flexible disk FD. When the program is recorded on theflexible disk FD, the computer system Cs writes in the moving picturecoding and decoding methods as the program via a flexible disk driveFDD. When the moving picture coding and decoding methods are constructedin the computer system using the program on the flexible disk, theprogram is read out from the flexible disk and then transferred to thecomputer system by the flexible disk drive FDD.

The above explanation is made on an assumption that a recording mediumis a flexible disk, but the same processing can also be performed usingan optical disk. In addition, the recording medium is not limited to aflexible disk and an optical disk, but any other medium such as aCD-ROM, a memory card and a ROM cassette capable of recording a programcan be used.

Sixth Embodiment

The following is a description for the applications of the movingpicture coding and decoding methods illustrated in the above-mentionedembodiments 1-5 and a system using them.

FIG. 31 is a block diagram showing an overall configuration of a contentsupply system ex100 for realizing content delivery service. The area forproviding communication service is divided into cells of desired size,and cell sites ex107˜ex110, which are fixed wireless stations, areplaced in respective cells.

This content supply system ex100 is connected to apparatuses such as acomputer ex111, a Personal Digital Assistant (PDA) ex112, a cameraex113, a cell phone ex114 and a cell phone with a camera ex115 via, forexample, Internet ex101, an Internet service provider ex102, a telephonenetwork ex104, as well as the cell sites ex107˜ex110.

However, the content supply system ex100 is not limited to theconfiguration shown in FIG. 31 and may be connected to a combination ofany of them. Also, each apparatus may be connected directly to thetelephone network ex104, not through the cell sites ex107˜ex110.

The camera ex113 is an apparatus capable of shooting video such as adigital video camera. The cell phone ex114 may be a cell phone of any ofthe following system: a Personal Digital Communications (PDC) system, aCode Division Multiple Access (CDMA) system, a Wideband-Code DivisionMultiple Access (W-CDMA) system or a Global System for MobileCommunications (GSM) system, a Personal Handyphone System (PHS), or thelike.

A streaming server ex103 is connected to the camera ex113 via thetelephone network ex104 and also the cell site ex109, which realizes alive distribution or the like using the camera ex113 based on the codeddata transmitted from the user. Either of the camera ex113, the serverwhich transmits the data and the like may code the data. The movingpicture data shot by a camera ex116 may be transmitted to the streamingserver ex103 via the computer ext111. In this case, either the cameraex116 or the computer ex111 may code the moving picture data. An LSIex117 included in the computer ex111 and the camera ex116 performs thecoding processing. Software for coding and decoding a moving picture maybe integrated into any type of recording medium (such as a CD-ROM, aflexible disk and a hard disk) that is a recording medium which isreadable by the computer ext111 or the like. Furthermore, a cell phonewith a camera ex115 may transmit the moving picture data. This movingpicture data is the data coded by the LSI included in the cell phoneex115.

The content supply system ex100 codes contents (such as a music livevideo) shot by a user using the camera ex113, the camera ex116 or thelike in the same way as shown in the above-mentioned embodiments 1-5 andtransmits them to the streaming server ex103, while the streaming serverex103 makes stream delivery of the content data to the clients at theirrequests. The clients include the computer ex111, the PDA ex112, thecamera ex113, the cell phone ex114 and so on capable of decoding theabove-mentioned coded data. In the content supply system ex100, theclients can thus receive and reproduce the coded data, and can furtherreceive, decode and reproduce the data in real time so as to realizepersonal broadcasting.

When each apparatus in this system performs coding or decoding, themoving picture coding apparatus or the moving picture decoding apparatusshown in the above-mentioned embodiments can be used.

A cell phone will be explained as an example of such apparatus.

FIG. 32 is a diagram showing the cell phone ex115 using the movingpicture coding and decoding methods explained in the above-mentionedembodiments. The cell phone ex115 has an antenna ex201 for communicatingwith the cell site ex110 via radio waves, a camera unit ex203 such as aCCD camera capable of shooting moving and still pictures, a display unitex202 such as a liquid crystal display for displaying the data such asdecoded pictures and the like shot by the camera unit ex203 or receivedby the antenna ex201, a body unit including a set of operation keysex204, a voice output unit ex208 such as a speaker for outputting voice,a voice input unit ex205 such as a microphone for inputting voice, arecording medium ex207 for recording coded or decoded data such as dataof moving or still pictures shot by the camera, data of received e-mailsand that of moving or still pictures, and a slot unit ex206 forattaching the recording medium ex207 to the cell phone exll5. Therecording medium ex207 stores in itself a flash memory element, a kindof Electrically Erasable and Programmable Read Only Memory (EEPROM) thatis a nonvolatile memory electrically erasable from and rewritable to aplastic case such as an SD card.

Next, the cell phone ex115 will be explained with reference to FIG. 33.In the cell phone ex115, a main control unit ex311, designed in order tocontrol overall each unit of the main body which contains the displayunit ex202 as well as the operation keys ex204, is connected mutually toa power supply circuit unit ex310, an operation input control unitex304, a picture coding unit ex312, a camera interface unit ex303, aLiquid Crystal Display (LCD) control unit ex302, a picture decoding unitex309, a multiplexing/demultiplexing unit ex308, a read/write unitex307, a modem circuit unit ex306 and a voice processing unit ex305 viaa synchronous bus ex313.

When a call-end key or a power key is turned ON by a user's operation,the power supply circuit unit ex310 supplies the respective units withpower from a battery pack so as to activate the digital cell phone witha camera ex115 as a ready state.

In the cell phone ex115, the voice processing unit ex305 converts thevoice signals received by the voice input unit ex205 in conversationmode into digital voice data under the control of the main control unitex311 including a CPU, ROM and RAM, the modem circuit unit ex306performs spread spectrum processing for the digital voice data, and thecommunication circuit unit ex301 performs digital-to-analog conversionand frequency conversion for the data, so as to transmit it via theantenna ex201. Also, in the cell phone exll5, the communication circuitunit ex301 amplifies the data received by the antenna ex201 inconversation mode and performs frequency conversion and theanalog-to-digital conversion to the data, the modem circuit unit ex306performs inverse spread spectrum processing of the data, and the voiceprocessing unit ex305 converts it into analog voice data so as to outputit via the voice output unit ex208.

Furthermore, when transmitting an e-mail in data communication mode, thetext data of the e-mail inputted by operating the operation keys ex204of the main body is sent out to the main control unit ex311 via theoperation input control unit ex304. In the main control unit ex311,after the modem circuit unit ex306 performs spread spectrum processingof the text data and the communication circuit unit ex301 performs thedigital-to-analog conversion and the frequency conversion for the textdata, the data is transmitted to the cell site ex301 via the antennaex201.

When picture data is transmitted in data communication mode, the picturedata shot by the camera unit ex203 is supplied to the picture codingunit ex312 via the camera interface unit ex303. When it is nottransmitted, it is also possible to display the picture data shot by thecamera unit ex203 directly on the display unit ex202 via the camerainterface unit ex303 and the LCD control unit ex302.

The picture coding unit ex312, which includes the moving picture codingapparatus as described for the present invention, compresses and codesthe picture data supplied from the camera unit ex203 using the codingmethod employed by the moving picture coding apparatus as shown in theembodiments mentioned above so as to transform it into coded image data,and sends it out to the multiplexing/demultiplexing unit ex308. At thistime, the cell phone ex115 sends out the voice received by the voiceinput unit ex205 during the shooting with the camera unit ex203 to themultiplexing/demultiplexing unit ex308 as digital voice data via thevoice processing unit ex305.

The multiplexing/demultiplexing unit ex308 multiplexes the coded imagedata supplied from the picture coding unit ex312 and the voice datasupplied from the voice processing unit ex305, using a predeterminedmethod, then the modem circuit unit ex306 performs spread spectrumprocessing of the multiplexed data retained as a result of themultiplexing, and lastly the communication circuit unit ex301 performsdigital-to-analog conversion and frequency transform of the data for thetransmission via the antenna ex201.

As for receiving data of a moving picture file which is linked to a Webpage or the like in data communication mode, the modem circuit unitex306 performs inverse spread spectrum processing for the data receivedfrom the cell site ex110 via the antenna ex201, and sends out themultiplexed data retained as a result of the inverse spread spectrumprocessing.

In order to decode the multiplexed data received via the antenna ex201,the multiplexing/demultiplexing unit ex308 demultiplexes the multiplexeddata into a coded stream of image data and that of voice data, andsupplies the coded image data to the picture decoding unit ex309 and thevoice data to the voice processing unit ex305, respectively via thesynchronous bus ex313.

Next, the picture decoding unit ex309, including the moving picturedecoding apparatus as described in the present invention, decodes thecoded stream of the image data using the decoding method correspondingto the coding method as shown in the above-mentioned embodiments togenerate reproduced moving picture data, and supplies this data to thedisplay unit ex202 via the LCD control unit ex302, and thus the imagedata included in the moving picture file linked to a Web page, forinstance, is displayed. At the same time, the voice processing unitex305 converts the voice data into analog voice data, and supplies thisdata to the voice output unit ex208, and thus the voice data included inthe moving picture file linked to a Web page, for instance, isreproduced.

The present invention is not limited to the above-mentioned system sinceground-based or satellite digital broadcasting has been in the newslately and at least either the moving picture coding apparatus or themoving picture decoding apparatus described in the above-mentionedembodiments can be incorporated into a digital broadcasting system asshown in FIG. 34. More specifically, a coded stream of video informationis transmitted from a broadcast station ex409 to or communicated with abroadcast satellite ex410 via radio waves. Upon receipt of it, thebroadcast satellite ex410 transmits radio waves for broadcasting. Then,a home-use antenna ex406 with a satellite broadcast reception functionreceives the radio waves, and a television (receiver) ex401 or a Set TopBox (STB) ex407 decodes a coded bit stream for reproduction. The movingpicture decoding apparatus as shown in the above-mentioned embodimentscan be implemented in the regenerating apparatus ex403 for reading outand decoding the coded stream recorded on a recording medium ex402 suchas a CD and a DVD. In this case, the reproduced moving picture signalsare displayed on a monitor ex404. It is also conceivable to implementthe moving picture decoding apparatus in the STB ex407 connected to acable ex405 for a cable television or the antenna ex406 for satelliteand/or ground-based broadcasting so as to reproduce them on a monitorex408 of the television ex401. The moving picture decoding apparatus maybe incorporated into the television, not in the Set Top Box. Also, a carex412 having an antenna ex411 can receive signals from the satelliteex410 or the cell site ex107 for replaying a moving picture on a displaydevice such as a car navigation system ex413 set in the car ex412.

Furthermore, the moving picture coding apparatus as shown in theabove-mentioned embodiments can code picture signals and record them onthe recording medium. As a concrete example, a recorder ex420 such as aDVD recorder for recording picture signals on a DVD disk ex421, a diskrecorder for recording them on a hard disk can be cited. They can berecorded on an SD card ex422. When the recorder ex420 includes themoving picture decoding apparatus as shown in the above-mentionedembodiments, the picture signals recorded on the DVD disk ex421 or theSD card ex422 can be reproduced for display on the monitor ex408.

As for the structure of the car navigation system ex413, the structurewithout the camera unit ex203, the camera interface unit ex303 and thepicture coding unit ex312, out of the components shown in FIG. 33, isconceivable. The same applies for the computer ex111, the television(receiver) ex401 and others.

In addition, three types of implementations can be conceived for aterminal such as the cell phone ex114: a sending/receiving terminalimplemented with both an encoder and a decoder, a sending terminalimplemented with an encoder only, and a receiving terminal implementedwith a decoder only.

As described above, it is possible to use the moving picture coding anddecoding methods described in the above-mentioned embodiments 1-5 forany of the above-mentioned apparatuses and systems, and by using thesemethods, the effects described in the above-mentioned embodiments can beretained.

From the invention thus described, it will be obvious that theembodiments of the invention may be varied in many ways. Such variationsare not to be regarded as a departure from the spirit and scope of theinvention, and all such modifications as would be obvious to one skilledin the art are intended for inclusion within the scope of the followingclaims.

INDUSTRIAL APPLICABILITY

The moving picture coding apparatus according to the present inventionis practical with the use of, for example, a personal computer equippedwith a communication function, PDA, a broadcasting station for digitalbroadcasting and a cell phone.

Also, the moving picture decoding apparatus according to the presentinvention is practical with the use of, for example, a personal computerequipped with a communication function, PDA, STB for receiving digitalbroadcasting and a cell phone.

1. A picture decoding apparatus for decoding a coded stream including amoving picture stream that has pictures of different sizes, withreference to a decoded picture, the apparatus comprising: a picturememory including a memory area for holding decoded pictures; a memorydivision unit operable to divide the memory area of the picture memoryinto division areas of a predetermined size; and a storage unit operableto sequentially store all data of one newly decoded picture insequential memory areas which are formed with one or more of thedivision areas of said picture memory.
 2. The picture decoding apparatusaccording to claim 1, further comprising a size change judgment unitoperable to judge whether or not a picture size has been changed bycomparing a picture size of a newly decoded picture and a picture sizeof a picture just decoded before, wherein in the case where it is judgedthat the picture size of the newly decoded picture has been changed,said memory division unit divides the memory area so that the size ofthe division area becomes the picture size of the newly decoded picture.3. The picture decoding apparatus according to claim 2, wherein saidstorage unit includes a first free area estimation unit for estimatingwhether or not at least one of the division areas is a free area; and inthe case where at least one of the division areas is estimated as a freearea, said storage unit sequentially stores all the data of the newlydecoded picture in the sequential memory area which is one of theestimated division areas.
 4. The picture decoding apparatus according toclaim 3, wherein said storage unit includes a first picture deletionunit operable to generate the free area by sequentially deleting apicture from the one whose display order is the oldest among the decodedpictures which are already held in said picture memory, and in the casewhere none of the division areas is estimated as a free area, saidstorage unit has the first picture deletion unit delete the decodedpictures until at least one of the division areas is estimated as a freearea.
 5. The picture decoding apparatus according to claim 4, whereinsaid storage unit further comprises a display possibility judgment unitfor judging an impossibility of the display when displaying the decodedpicture which is already held in said picture memory in the case wheresaid decoded picture is deleted before being displayed.
 6. The picturedecoding apparatus according to claim 2, wherein said storage unitincludes a division area selection unit for sequentially selecting thedivision area from the head of said picture memory, and said storageunit sequentially stores all the data of one newly decoded picture inthe sequential memory area which is formed with the selected divisionareas.
 7. The picture decoding apparatus according to claim 6, whereinsaid storage unit includes, a second free area estimation unit forestimating whether or not the selected division area is a free area; anda second picture deletion unit for making the selected division area afree area by deleting the decoded picture which is stored in thedivision area in the case where the selected division area is not a freearea, wherein said storage unit sequentially stores all the data of thenewly decoded picture in the sequential memory area which is formed withthe selected division areas.
 8. The picture decoding apparatus accordingto claim 3, wherein said storage unit includes, a division areaselection unit for sequentially selecting the division area from thehead of the picture memory in the case where none of the division areasis estimated as a free area; a second free area estimation unit forestimating whether or not the selected division area is a free area; anda second picture deletion unit for making the selected division area afree area by deleting the decoded picture which is already stored in thedivision area in the case where the selected division area is not a freearea, wherein said storage unit sequentially stores all the data of thenewly decoded picture in the sequential memory area which is formed withthe selected division areas.
 9. The picture decoding apparatus accordingto claim 1, further comprising a reading unit operable to read outdivision information indicating a division method of the memory areafrom the coded stream, wherein said memory division unit divides thememory area according to the read division information.
 10. The picturedecoding apparatus according to claim 9, wherein the divisioninformation indicates the biggest picture size among the moving picturestreams included in the coded stream, and said memory division unitdivides the memory area so that the size of said each division areabecomes the picture size indicated by the division information.
 11. Thepicture decoding apparatus according to claim 9, wherein the divisioninformation indicates the number of the division areas in the case wherethe size of the memory area is divided so that the size of the dividedmemory area is equal to the biggest picture size, and said memorydivision unit divides the memory area into the number of the divisionareas.
 12. The picture decoding apparatus according to claim 9, whereinthe division information indicates the smallest picture size among themoving picture streams included in the coded stream, and said memorydivision unit divides the memory area so that the size of said eachdivision area becomes the picture size indicated by the divisioninformation.
 13. The picture decoding apparatus according to claim 9,wherein the division information indicates the number of the divisionareas in the case where the size of the memory area is divided by thesmallest picture size, and said memory division unit divides the memoryarea into the number of the division areas.
 14. The picture decodingapparatus according to claim 9, wherein the division informationindicates the greatest common divisor of the picture sizes of the movingpicture streams included in the coded stream, and said memory divisionunit divides the memory area so that the size of said each division areabecomes the picture size indicated by the division information.
 15. Thepicture decoding apparatus according to claim 9, wherein a valueindicated by the division information is rounded up so that at least thesame macroblock is stored within said same division area.
 16. Thepicture decoding apparatus according to claim 9, wherein a valueindicated by the division information is rounded up so that all themacroblocks of one row in horizontal direction in said each decodedpicture are stored in said one division area.
 17. The picture decodingapparatus according to claim 9, wherein said memory division unit roundsup the size of the division area so that at least the same macroblock isstored within said same division area.
 18. The picture decodingapparatus according to claim 9, wherein said memory division unit roundsup the size of the division area so that all the macroblocks equivalentto one row, in horizontal direction, in said each decoded picture arestored in said one division area.
 19. The picture decoding apparatusaccording to claim 9, wherein the division information further includesinformation indicating whether or not there is a possibility that thedecoded picture already stored in the picture memory may be deletedbefore being displayed.
 20. The picture decoding apparatus according toclaim 9, wherein said storage unit includes the first free areaestimation unit for estimating whether or not at least one of thedivision areas within the picture memory is a free area, and in the casewhere a sequential memory area formed with one or more of said divisionarea is estimated as a free area, said storage unit sequentially storesall the data equivalent to said one newly decoded picture in thesequential memory area formed with said estimated division area, saidsequential memory area being capable of sequentially holding all thedata equivalent to said one newly decoded picture.
 21. The picturedecoding apparatus according to claim 20, wherein said storage unitincludes the first picture deletion unit operable to generate the freearea by sequentially deleting the picture from the one whose displayorder is the oldest among the decoded pictures stored already in thepicture memory, and in the case where the division area is not estimatedas a free area, said storage unit has the first picture deletion unitdelete the decoded picture until said division area is estimated as afree area, said division area being capable of sequentially holding allthe data of said one newly decoded picture.
 22. A picture codingapparatus for generating a coded stream by coding a moving picturestream that includes pictures of different sizes, with reference tocoded pictures, the apparatus comprising: a picture size acquisitionunit operable to acquire a picture size of a moving picture streamincluded in a generated coded stream; a division information generationunit operable to generate division information indicating a divisionmethod of a picture memory on the picture decoding apparatus side, basedon the acquired picture size; and a coding unit operable to generate acoded stream including the generated division information.
 23. Thepicture coding apparatus according to claim 22, wherein the divisioninformation generation unit generates the division informationindicating the biggest picture size among the acquired picture sizes.24. The picture coding apparatus according to claim 22, wherein thedivision information generation unit generates the division informationindicating the smallest picture size among the acquired picture sizes.25. A picture decoding method for storing a decoded picture in a picturememory and decoding a coded stream including a moving picture streamthat has pictures of different sizes, with reference to said decodedpicture, comprising: a memory division step for dividing the memory areaof the picture memory into division areas of a predetermined size; and astorage step for sequentially storing all data of one newly decodedpicture in the sequential memory areas which are formed with one or moredivision areas of the picture memory.
 26. The picture decoding methodaccording to claim 25, further comprising a size change judgment stepfor judging whether or not the picture size has been changed bycomparing the picture size of a newly decoded picture and the picturesize of a just decoded picture, wherein in the case where it is judgedthat the picture size of the newly decoded picture has been changed, thememory division step divides the memory area so that the size of thedivision area becomes the picture size of the newly decoded picture. 27.The picture decoding method according to claim 26, wherein the storagestep includes a first free area estimation step for estimating whetheror not at least one of the division areas is a free area; and in thecase where at least one of the division areas is estimated as a freearea, the storage step sequentially stores all the data of one newlydecoded picture in the sequential memory area which is one of theestimated division areas.
 28. The picture decoding method according toclaim 27, wherein the storage step includes a first picture deletionstep for generating the free area by sequentially deleting a picturefrom the one whose display order is the oldest among the decodedpictures which are already stored in the picture memory, and in the casewhere none of the division areas is estimated as a free area, thestorage step has the first picture deletion step delete the decodedpictures until at least one of the division areas is estimated as a freearea.
 29. The picture decoding method according to claim 26, wherein thestorage step includes a division area selection step for sequentiallyselecting the division area from the head of the picture memory, and thestorage step sequentially stores all the data of the newly decodedpicture in the sequential memory area which is formed with the selecteddivision areas.
 30. The picture decoding method according to claim 29,wherein the storage step includes, a second free area estimation stepfor estimating whether or not the selected division area is a free area;and a second picture deletion step for making the selected division areaa free area by deleting the decoded picture which is stored in thedivision area in the case where the selected division area is not a freearea, wherein the storage step sequentially stores all the data of thenewly decoded picture in the sequential memory area which is formed withthe selected division areas.
 31. A picture coding method generating acoded stream by coding a moving picture stream that includes pictures ofdifferent sizes, with reference to coded pictures, comprising: a picturesize acquisition step for acquiring a picture size of a moving picturestream included in a generated coded stream; a division informationgeneration step for generating division information indicating adivision method of a picture memory on the picture decoding apparatusside, based on the acquired picture size; and a coding step forgenerating a coded stream including the generated division information.32. A program for a picture decoding apparatus storing decoded picturesin a picture memory and decoding a coded stream including a movingpicture stream that has pictures of different sizes, with reference to adecoded picture, having a computer perform following steps: a memorydivision step for dividing the memory area of the picture memory intodivision areas of a predetermined size; and a storage step forsequentially storing all data of one newly decoded picture in thesequential memory areas which are formed with one or more division areasof the picture memory.
 33. A program for a picture coding apparatusgenerating a coded stream by coding a moving picture stream thatincludes pictures of different sizes, with reference to a coded picture,having a computer perform following steps: a picture size acquisitionstep for acquiring a picture size of a moving picture stream included ina generated coded stream; a division information generation step forgenerating division information indicating a division method of apicture memory on the picture decoding apparatus side, based on theacquired picture size; and a coding step for generating a coded streamincluding the generated division information.